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ABSTRACT 

This 5-section report summarizes the results of a 
project concerned with how novices learn to' become creative 
educational computer users. Based on a cognitive analysis of 
elementary programming statements in BASIC and calculator language 
into conceptual units, the project builds on previous research on 
learning BASIC programming. A general description of the project. is 
followed by a review of the literature on the psychology of computer 
programming, especially research in cognitive psychology related to 
beginning programming. The analysis of both a calculator language and 
BASIC into conceptual units and the evaluation of users' knowledge is 
then described. Data on the frequency, of misconceptions in users' 
knowledge of calculator and BASIC languages are presented. Focusing 
on instructional techniques useful in the remediation of specific 
misconceptions of users' knowledge of calculator and BASIC languages, 
the final chapters explore the effectiveness of teaching mental 
models and whether use of the models enhances problem-solving 
performance. References are included in each section and. an appendix 
lists additional papers and publications which are prbducts of the 
project under review (either published or in preparation). (LMM) 
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CHAPTER 1 
IIITRODUCTION 



-'2 

. Focus of Pro.iect 

This project is concerned with how novices learn to become creative users 
of electronic cooiputers. In particular, this project contributes to what 

^ • • ' » 

Scheiderman (1980) calls "software psychology" — a theory of how humans think 
about and" use computers. This project is based on a cognitive analysis of 
elementary progfammlng statements in BASIC and Calculator Language into 
conceptual units, similar to techniques for, analyzing arithmetic computational 
skill (Bi;own 4 Burton, 1978; Groen & Parkman, 1972; Resnick, 1976). In 
addition, this project builds on previous research on learning BASIC computer 
programming (Mayer, 1979). . • . " . * 

Or. 

The malti goal's of J;he project are the following: 

(1') Analysis of the "bugs" in people's knowledge of elementary computer 

programming. This project specifies a user's l^wledge in terms of specific 

"bugs" in his or her understanding of each statement or command. By using a 

■ore detailed level of analysis than has be6n traditionally us.ed, we have been 

able to make specific diagnoses of what userS do not know. This technique is 

slmlW in some respects to that used by Brown and Burton (1978) in analyzing 

I 

procedural "bugs" in arithmetic* ^ ^ ' ,j 

(2) Remediation of common "bugs" in computer knowledge -^through 
Instruction, Once a user's knowledge Is desci^lbed In terms of "bugs" In his 
or her understanding of statements, lt*ls poWible to develop remedial 
training that focuses, on the missing conce^^ts. As part of remedial 
Instruction, we have used concrete models of the computer which allow us to 
show the flcjw of information for eacH^ line of code in a concrete and visual 
way. Our instruction helps ^the users to see what goes on "inside the 
computer" for each statement. 

^ ' ■> 

(3) Determination of tfchritLques for helping students to become creative 
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users of computers. A third goal of this project is to determine whether the 
diagnosis and remediation techniques outlined above will enhance the 
problem-solving performance of our students. This project provides both a 
practical and a theoretical test of the usefulness >Df cognitive ana^lysis 
techniques in a real learning and problem-solving task. In part^icular, 
project hak demonstrated that instruction in specific "mental models" has a 
positive influence on problemr-solving performance in programming. 
Rationale of Project ^ 

Some knowledge of computer programming is rapidly vbecoming a survival 
skill. Elementary computer programming is being incorporated into science and 
mathematics curricula of many schools, and as part of the technical traini^ig 
for many adults. Recently, Carnegie-Mellon University has announced that all 
students will be required ^o buy computer terminals as part of their college 
educational materials. With increasing improvements in computer power, 
coupled with decreasing costs, there are indications that computers will 
become* a common feature of our nation's classrooms. 

• In spite of tremendous^ achievements in hardware development and the 
advances in producing educational- software, there has been comparatively 
little basic research on how to teach computer programming. . Other topics in 
mathematics and science have enjoyed long histories of instructional research, 

but computer programming is so new that it has not enjoydd a similar research 

^ ) \ 

emphasis. ' 

L 

Since computers are becoming a part of many classrooms in our nation, and 
' Fince elementary programming is becoming ^'a part of > .math and science 
curriculum, there-<ts a need to better understand how beginners learn to use 
computers." This project provides new information concerning the processes by 
which novices learn computer programming, and how to help novices become 
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productive users of computfers. 
Organization of the Final Report 

This Final Report describes the methods and results of the project. ' 
There are five major sections of the report, covering each of the tasks listed 
in the project proposal. 

Chapter 2 pres^nt« a review of literature concerning the psychology of 
computer programming. The review focuses on research in cognitive psychology 
that is relevant for the study of how novices learn programming languages. 

Chapter 3 deocrlbes how a calculator language can be analyzed Into 
conceptual units and how users' knowledge can be evaluated. In addition. 
Chapter 3 presents data concerning the frequency of major "^gs" or 
misconceptions in users' knowledge of calculators. 

Chapter 4 describes how BASIC can be analyzed into conceptual units and 
how'users' knowledge can be evaluated. In addition. Chapter 4 presents data 
concerning the frequency of major "t)ugs" (or misconceptions) in users' 
, knowledge of BASIC. 

Chapter. 5 focuses on instructional techniques that may be useful in the, 
remediation of specific "bugs" in users' knowledge of calculator language. In 
» particular, this chapter explores whether students can be taught to use 
"mental models" and whether use of the models enhances problem-solving 
performance.* 

Chapter 6 focuses on" instructional techniques that may be useful in the 
remediation of user'? "bugs" concerning BASIC statements. In particular, this 
chapter explores whether students can be taught to use various "mental models" 
for, BASIC, and whether use of the models enhances problem-solving perfoirmance. 

^ c 

These six chapters constitute , the first volume of this final report. The 
second volume- summarizes the shme research, but does so in a way more 
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appropriate for practitioners. * • . - 

Project Accomplishments ^ ' 

. The Major accomplishments of this project have been th^ following: 
Task analysis . We have developed and iseflned an analysis of BASIC and 
Calculator Language. The analysis yields a set of elementary actions that go 
on inside the computer, and which are Important for the user's understanding 
of the language. " 

Development of Evaluation Instruments . We developed ^ structured ' 
questionnaires to assess a user s knowledge of the staten\(ents of BASIC and 
Calculator Language. Through several studies, ^ we refined, the evaluation 
instruments so that we are now able to measure user's conceptions of computer 
languages. ^ 

Diagnosis of bugs . Using the evaluation instruments, ^e have identified^ 
typical bugs in users' understanding of BASIC and Calculator Language. For 
example, a high proportion of users think that LET X « A + B means that an 
equation Is 8t6red in memory. As^anqther example, a high proportion of tlsers 
^ think , that READ A means that the computer prints out the value stored in 
memory space A. A majpr product of this diagnosis effort is a list of 
standard misconceptions for each statement, based on student performance. 

Remediation of bugs . We have developed and Implemented trainii)^g 
procedures that are aimed at corrfecting specific bugs •in a user's knowledge. 
For both BASIC and Calculator Language, we have developed instructional' 
techniques based on our task analysis. These techniques have strong and 
reliable effects on elimination of bugs in user's ^ndetstanding of statements. 

Effects on problem solving . Finally, w^ have studied the effects of 
user's ^knowledge of BASIC on actual problem solving in BASIC. The results 

m 

Indicate that students' problem-solving performance is enhanced when bugs in 
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un4MS?andlng of- statements are minimal. Thus, .we have bttained evidence that 

user's knowledge of BASIC statements at a "transactioTr level" is related to- 

spperior problem^solving performance. 

Project Implications 

This project provides new information concerning how students learn 

* 

computer programming statements. In particular, this project has documented 

« *• 

the existence of users* miscprig^ptions of computer p^gramming statements, a«id 
has exatjined ins^tructional techniques for rem^ediating these misc(inceptions. 

The results have implications both for cognitive theory and for^ the teaching 

/ 

of ^Gomputer programming in American schools. 

^* « ' 

The results indicate thaC**»"hands— on experieuce" does not necessarily lead 
to understanding of programming languages. Many situdents developed sferious ' 
bugs* in spite of hands-on experience. Som^ direct instruction is required * 
concerning the underlying conceptual events and ideas ^^n computer programming. 

The results indicate that it is possible to provide direct? Instruction 
concerning 'what goes on inside the computer" for each statement,.. This , level 
of " instruct ipn is needed when the goal of Instruc^on is creative problem 
solving. 

The "^ipelKilts indicate that remediation efforts could b*e direfcted at* 
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specific bugs rather than overall mastery. 

"What is learned" in computer programming^ involves both specific facts 
and a "mental model" of the system. • Students, need help in developing 
productive mental models, including direct instructlQn ,about models. 
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INTRODUCTION , 

This article focuses on the qu^stion^ "What have we .learned about how -to 
Increase the novice's understanding of computers and computer programming?" In 
particular, this paper reviews ideas from cognitive and educational psychology 
that are related to the problem of how to teach non-programmers to use com- 
puters.^. Since people who are not professional programmers will have to learn 
how to Inte^ract with computers, an important issue concerns how to foster 
meaningful learning of -computer concepts by novices. 

Meaningful learning is viewed as a process in which the learner connects 
new material with knowledge that already exists in memory (BRAIJ79) . The 
* isting knowledge in memory has been called "schema" and the process of con- ' 

dh ' > ' 

nectlng new information to it has been called "assimilatlpn. " However, thera 
is not *yet agireeiiient concerning the specific mechanisms that are involved in 
'"^similation to schema" (ANDE77, AUSU77, BARD32, KINT74, MINS75, RUME75,^ 



SCHA7^ THOR77) . ' ^ " . 

Figure 1 provides a general f rameworkj for discussion the process of 
^ meaningful learning (or assimilation to schema) of technical information 
*(MAYE75a, MAYE79a) . In the figure, the human cognitive system is broken down 
into: 4 

short term memory — a temporary and limited capacity store for holding 
I <«nd manipulating information, / and 

long term memory — a permanent, <Jrgantzed, and unlimited store of 
existing kiiowledge. 

New technical' information enters the human. cognitive system from the outside 

* 

and must go through the^ following steps for meaningful learning to occur^: 



ERIC 



(a) Reception , First the learner must pay attention to the Incoming 
Information so that it reaches short tarm memory (as indicated by 
arrow a) * 

(b) Availability , Second, the learner must possess appropriate 
prerequisit^e concepts in long term memory to use in assimilating 
th€i new information (as indicated by point b.) ^^"^ 

(c) Activation , Finally, the learner must actively use tl, s pre- 
requisite knowledge during , learning so that the 'new material 

* maybe connected with it (as indicated by arrow c ^rom long term ^ 
memory to short term memory). 
Thus, in the course of meaningful learning, the learner must come into 
contact td.th the new material (by bringing it into working memory), then must 
search long term memory for what Ausubel (AUSU68) calls "appropriate anchoring 
ideas" or "ideational scaffolding," and then mu6t transfer Xhose ideas to 
working memory so they can be combined with new incoming information. If , any of 
these conditions is not met, meaningful learning cannot occur; and the learner 
will forced to rotely memorize each piece of new information as a separate 
item to be added to memory. The techniques reviewed in this article are aimed 
at ins^ring that the availability and activation condition^ are likely to be 
met , > 



Inrert Figure 1 about here 



The goal of this article is to explore techniques for Increasing the 
novice's understanding of computer programming by exploring techniques that ^ 
activate the "appropriate anchoring ideas." Two techniques reviewed in this 
pap^^;^: (1) providing a familiar concrete model of the computer, and 



•(2) encouraging learners to put technical information into their own words. 
Each technique is an attempt to foster the process by which familiar existing 

knowledge is connected with new incoining techn'^.cal information. For e^cb 

• « 

technique, a brief rat ion ale is presented, examples of research are presented, 
and ah evaluative summary is offered, 

1. UNDERSTANDING OF TECHNICAL INFORMATION BY NOVICES 
1.1 Definitions 

For present purposes, understanding, is defined as the ability to use 
learned information ia problem solving tasks that are different from what was 
explicitly taught. Thus, understanding is manifested in the user's ability to 
transfer learning to new situations* Novices are defined as users who have had 
little or no previou's experience with computers and who do not intend to become 
professional programmers, and thus lack specific knowledge of computer pro- 
grai^ming. 

1>2 Distinction Between Understanding and Rote Learning 

The Gestalt psychologists (WERT59, KAT042, KOHL25) distinguished between" 
. two ways of learning how to solve problems — "rote learning" vs* "tinder standing*' 
-With respect to mathematics learning, for example, there is often a distinction 

0 

made between "getting the right answer" and "understanding what you a^e doing*" 
In a classic example, Wertheimer suggests |:hat there are two basic ways t6 
td^ch a child how to f Aid the area of a par^lelogram (WERT59) . One . method 
involves dropping a perpendicular line, measuring the height of the perpen- 
dicular, measuring the length of the base, and calculating area by use of the 
formula. Area = Height x Base* Wertheimer calls this the "rote learning" or 
^"senseless" method, because the stv^dent simply memorizes a formula and a pr<^ 



cedure* The other method calls ftfPche student to visually explore the par- 
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-illelogram until the student sees that you could cut a triangle from one end. 
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put it on the other end, and form a rectangle. Since the student already knows 
how to find the area of a rectangle, the problem is solved. Werthelmer calls 
this method "structural understanding" or "meaningful apprehension of relations," 
since the learner has gained insight into the structure of parallelograms. % 

According to Werthelmer, if you give sa test involving parallelograms like 
the one used during instruction, both groups of children will perform well. 
However, if you give a transfer test that involves urilisual parallelograms then 
the rote learners will say, "We haven't had this yet," while the understanders 
will be able to derive answers. Thus, the payoff for understanding comes not in 
, direqt application of the newly learned material, but rather in transfer to new 
situations. This example suggests that when creative use*' of new technical 
information is the goal, it is important to use methods 'that foster under- 
standing. 

2.0 DO CONCRETE MODELS AID MEANINGFUL LEARNING OF COMPUTER PROGRAMMING? 

2.1 Statement of the Problem 

Since novices lack domain specific Knowledge, one technique for improving 
their understanding of new technical information is to provide them with a 
framework that can be used for incorporating new information. This technique 
is aimed at Insuring availability of knowledge in long term memory (See Figure 1). 
The present section explores the effects of concrete models oii people's under- 
standing and learning of new technical information such as computer programming. 
The major research questions concern how concrete models influence the learning 
process and how to choose an effective model. 

2.2 Concrete Models in Mathematics Learning < 

One technique for providing the appropriate prerequisite knowledge is the 
use of familiar, concrete models. For example, Brownell & Moser (BROW49) 
taught third graders how to use a subtraction algorithm, using two different 
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methods. One group of several hundred children was taught by using concrete 
objects like bundles of sticks. For these children, concepts like "borrowing" 
and "place value" were described in terms of rearranging bundles of sticks into 
groups of tens. The other group^ was-^^ghtr^n a *^pur^^ 

fashion"; these children were explicitly given the rules for subtraction at the 
start and given plenty of "hands on" experience in executing the procedures on 
standard two digit subtraction problems. Although both groups of students 
learned to perform equally well on standard two digit subtraction problems, the 
students who learned with bundles of sticks performed better on tests in- 
volving transfer problems (e.g., more complicated subtraction problems). 

In current instructional practice, manipula'tives , such as ^oins or sticks 
or blocks, are used in mathematics teaching in order to make computational 
procedures more concrete (WEAV72, RESN80). In a careful set of interviews with 
children who were learning to subtract, Resnick & Ford (RESN80) noted that 
children often invented a concrete model to help them understand the procedure. 
Since computer programming shares many of the characteristics of computational 
procedures in mathematics, it seems possible that the use of manipulatives in 
computer programming might be as successful as in mathematics. 
2.3 Models, Titles, and Advance Organizerg in Text 

There is also encouraging evidence tll^t similar techniques may be used to 
increase th<^ meaning fulness of technical information presented in text. For 
example, BranSford & Johnson (BMN72) presented the following passage to sub- 
jects: 

The procedure is actually quite simple. First you arrange 

items into different groups. Of course, one pile may be ^ 

sufficient depending on how much there is to do. If you 
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have to go somewhere else' due to lack of facilities that is 
the next step; othe^ise, you are pretty well s«t. It is 
important not to overdo things. In the short run this may 
not seem important, but complications can easily arise. A 
mistake can be expensive as well. At first, the whole 
procedure will seem complicated. Soon, however, it will 
become just another facet of life. It is difficult 
to foresee any end to the necessity for this task in 
* the Immediate future, but then, one never can tell. 
After the procedure is completed one arranges the 
materials into different groups again. Then they can 
be put into their appropriate places. Eventually 
^ they will be used once more and the whole cycle will \ 
have to be repeated. However, this is part of life. 

Subjects who read this passage, without a title, rated it low in compre- 
hensibility (2.3 on a 7 point scale) and recalled an average of only 2.8 out of 
18 ideas from the passage. However, some subjects were given a description of 
the topic — washing clothes — before the passage. These subjects rated the 
passage much higher in comprehensibility (4.5 on^a i point scale) and recalled 
more than twice as much information (5.8 idea units out of 18). In addition, a 
third gtoup was given the washing clothes topic after the passage was presented. 
However, this group performed at about the same low level as the subjects who 
Vere givea no topic, '(rating the passage at 2A/±n comprehension and recalled an 
average of 2.7 idea units). Similar studies (BRAN72, D00L71, DOOL72) also found 
that students* recall of Ambiguous and technical passage? was enhanced when an 
organizing title or diagram or sentence was given V^ior to reading. However, 
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these techniques did not have the sam^ f aoljQs^tating effect when presented after 
the student had read the passage. These results suggest that the learner must 
have an appropriate asslnjilative set available at the time of learning. Even 

though the same total amount of information may be presented, the students' 

- ^ f ■ ' # 

ability to recall and use.the information in the passage is much higher when 
the clarifying title or picture is given before rather than after reading. 

Ausubel (AUSU68) has argued that learning of' new technical prose may be 
enhanced by providing an advance organizer — a short expository introduction, 
presented prior to the text, containing no specific content from the text, but 
providing the general concepts and ideas that can be used to subsume the in- 
formation in the text. The first advance organizer studies conducted by 
Ausubel and his colleagues in the early 1960's (AUSU60, AUSU63, AUSU68), 
provided some support for this assertion. For example, in a typical study 
(AUSU60), 120 college students read a 250C»^word text oii metallurgy after 
reading either a 300-word advance organizer that presented the underlying 
framework for the information or a control 500-word historical passage. Tne 
^advance organizer presented the abstract principles involved in the text. On a 
reading comprehension post-test covering the basic information in the passage, 
the advance organizer, group performed significantly better than the control 
group, with scores of 47% C9rrect versus 40Z correct, respectively. 

More recently, reviews of the advance organizer literature reveal that 
advance organizers tend to have their strongest effects in situations where 
learners are unlikely to already possess useful prerequisite concepts — namely, 
for technical or unfamiliar material, for "low ability" or inexperienced' 
students, and when the test involves transfer to new situations (MAYE79a,, 
HA,YE79b). For example, to study the effects of advance organizers on different 
kinds of ^materials, Lesh (LESH76) asked 48 college students to watch a four- 

18 
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hour videotape on finite, geometry. An organizer that ~g5^«.-^ncrete exanvples 
and models^ was provided either before or after instruction. The instructional 
lesson \fks organized either in a order of increasflng difficulty (hierarchical - 
order) or in an order that repeated key concepts and related new toaiierial to 
previous material (spiral order) . Results af a standard post-test indicated 
that the advance organizer group outperformed the post-organizer group^or the 
hierarchical unit, but the difference was much ^ess for the*spiral unit. ^ 
Simflar treatment x material type interactions were obtained using social 
studies lessons (SCHU75) and ''mathematics lessons (GR0T68) • Similarly, Raye 
(RAYE73) reported that the title biasing effects obtained by Bransford fit' 
Johnbon (BRi^75) with the washing clothes passage were eliminated when the 
passage was made more concrete and familiar. Thus, ther^ is consistent evi- 
dence that organizers have stronger effects for unfamiliar, abstract infor- 
mation than for familiar, concrete information. i - 

^ III a study invest igatir^ the effects of advance organizers on students 
with high and low ability (or knowledge), physics material was taught to high 
sdhool students (WEST76). Advance organizers, consisting of condrete models, 
tended to improve test performai^ca^f low ability * students but had a much 
smaller effect for hi^h ability subjects. Similar group x ability interactions 
were obtained by several other researchers (RING71, FITZ63, AUSU62, AUSU61, 
AUSU63, 'AUSU77, SMIT69) , Thus, there is evidence that advance organizers have 
a stronger effect on low knowledge or low ability learners as compared to high 
knowledge or high ability learners. 

Finally, in studies involving transfer tests (i.eo problems that are 
different from those in instruction), there is consistent evidence that advance 
organizers have a stronger effect on transfer performance thaij^on simple re- 
•tentlon. For example, this pattern was obtained with material on mathematical 
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topology (SCAN67J, number bases (GROj:68, MAYE77), and an imaginary science 
(MERR66). 

Many of the appareiitr conflicts in the adyance organizer literature 
(BARN75, LAWT77) can be accounted for by the idea that advance organizers 
find/a t^y of connect ing^ew- information with existing knowledge — organizers 
are not needed for familiar material, experienced learners, or when the test 
does not "^ involve jransfer. 

i 

While there is at present no foolproof procedure for generating useful 
advance organizers, a careful review of the existing literature suggests the 
following guidelines (MAYE79a) : (1) The* organizer should allow the reader to 
generate all or some of the logical relations in the text. (2) The organizer 
should provide a means of relating the information in the text with existing 
knowledge. (3) The organizer shou^'d be familiar to tfie learners. (A) The 
organizer encourages the learner to prerequisite knowledge that the learner 
would not normally have used. To date, advance organizers have been' most/ 
effectively used in mathematics and science topics (MAYE79a). 

Royer and his colleagues (ROYE75, ROYE76) have demonstrated that concrete 
Idbdels may serve as effective advance organizers In learning of new scientific 
information. In their .studies, subjects read two passages, such as a pai^sage 
on electrical conductivity followed by a passage on heat flow. For some \ • 



subjects, the first passage contained several concrete analogies, such asf 

• ' ' ^ ■ ' ' ' [^ 

electrical conduction. being dfiscrilied as a chain of falling dominoes. Fcnif' 

other 'subjects, the first passage presented the same information in abstract 

form without any concrete analogies. Reading of the second passage was facili- 

tated if students h^^ been given concrete models. In the first passage (e.g., 

recall- of the information In the second passage was about twice that of control 

groups). Apparently, the models presented in the first passage could be used 

20 
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by learners during the reading of the second passage help relate the tech- 

*■ » 

nlcal terns to familiar concepts. . 

. Similarly, White & Mayer OffllTSO) analyzed physics textbooks to determine 
how concrete models were used. For example, many textbooks explain Ohm's Law 
by describing water flowing in pipes, or a boy pushing a heavy load up an 
inclined street, ot electron flow through a circuit. Recent results (MAYE80) 
show that when concrete analogies are embedded in a technical textl novices 
tend to perform best on recalling these familiar models and tend to recognize 
the information adjacent to the model in the tfext.' 
y 2.h Concrete Models in Computer Programming 

In previous' sections, research was presented concerning the role of 
manipulatives in mathematics instruction, titles and pictures in remembering, \ 
ambigupus passages, and advance organizers and models in science text. In eaph 
case there was evidence tliat these techniques serve to provide the learner with 
appropriate anchoring knowledge that is required for comprehension of ^ew tech- 

nical information. The present section focuses on research related specif- 

i * * ' 

ically to computer ^rograito(ii.ng. 

DuBoulay and his collfiSgues (DUB076, DUBOSO) have provided a concrete 
model for teaching LOGO to children. The model consists of a conceptual LOGO 
machine with concrete memory locatiohs, switches and work space, which allow 
the learner to "work" the machine. 

DuBoulay and his collpague^ have argued that there are two basic approaches 
to learning to interact with a computer. The first approach could be called 
the black box approach .- In this approach the user deveiopes the attitude that . 
the computer is a black box — you put in cbmroands and data and out comes the 
answer as if by magic. The mechanisms by which the computer*^ operates are 
hidden from the user, and the user is likely to assume that computers are just 



not understandable. Such users are likely to memorize algorithms that "work" — 
i.e., that generate the desired answers. However, such ysers are not able to 
relate the commands to an understanding of what goes on inside the black box. 

The second approach is what can be called the glafes box approach * In this 
approach, the user attempts to understand what is going on Inside the computer. 
Each command results in some change in the computer and these changes can be 
described and understood. ' The level of description need not — indeed should 

. not — ^be at the "blood and guts" level. Users do' not need to become electronics 
experts . . There* is an appropriate level of description that Mayer (MAYE79c) 
refers to as the "transaction level." Similarly, DuBoulay et al; (DUB080) 
offer two important propertieis for making hidden operations t)f a language more 
clear to a novice: (1) "simplicity — there should be a, "small nupiber of parts 
.that interact in ways that can be easily understood,'! and (2) visibility — 
novices should be able to view "selected parts and processes" of the model 
"in action," The LOGO model appears to fit these specifications because it 
is a simple, familiaz^ modtel of the computer operations involvec} ih LOGO; in 
short, it allows the user to develop intuitions about what goe^ on inside th? 
computer for each line of code. Unfortunately, however, DuBoula^r and his 
colleagues have not provided empirical tests concerning whether the LOGO 
machine model actually influences the pro'^em solving performance of ne\f ' 

. learners, as compared to "traditional" methods that emphasise only "hands on 
experiences/" * ^ 

2.5 Effects of Models on Transfer Performance 

In order to provide some information concerning the effects of concrete 
modela' on learning computer programming, a . series of Studies was conducted 
(MMrE75b). In the studies, /subjects were either given a concrete model of' the 
ccM&puter or not. Then subjectstread a manual on a BASIG-rlike language and took 
a transfer test on the material. 



Method ; .Figure 2. shows the model of the computer that was used to explain 

» " / 

elementary BASIC-like statements to novices. The itodel .provides concrete 



Insert Figure 2 about /here. 



analogies for four major functional units of tlje computer: ^) input is repre- 
sented as a ticket window in which data is lin^kd up waiting to be processe ind* 
is placed in the finished pile after being processed, (2) output. is represented 

as a message no^e pad with one message writtejb per line» (3) memoiry is repre- 

, . . 'I » . 

sented as an eraeieable scoreboard in which tHere is natural destructive read- in 

and non-destructlv^ read-out, and (4) executjlve control is represented as a 
recipe or shopping list with a pointer arrow to indicate tl\e line being exe- 
cuted. This model is similar to DuBoulay *s jmodel of the LOGO machine in the 

^ . • • ' ' • : 

way it makes the basic operations of th^ computer visible to the learner. A 

<■• ■ 

2x3 foot diagram containing these parts^ anct a brief one page description 
were provided to subjects in the "mod^l gro&p" (see Figure 2) but no model was 
given to the "control group." ^ • 

All subjects then were given a l6-page manual that described seven state- 
ments modified from BASIC and FORTRAN, (see Table 1). For each statement » the 
i&anual presented the statement ,,'provlded the grammar rules for the statement 
(e.g.y definitions of legal address names) , and gave an example of the state^ 
ment as it might occur in a' line oj^a program. Subjects ii'i both groups were 
.given the saihe manual to read at their own rates, averaging 2Q^ to 30 minutes. 

Following reading, the same test was given to all subjects. The test 
consisted of six types of problems: (1) generate-statement problems gave a 
problem In English and reqqired a one statement program as the solution. 
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(2) generate-nonloop problems gave a problem in English and required a short 
non-looping program for solution, (3) generate-looping problems gavQ a problem 
in English and required a looping ^program for solution, (4) interpret-stagement 
probl'ei&s gave a single statement program and asked the student to describe what 
the computer would do, (5) lnterpfet-npnlo.pp gave^la non-looping program ^d 
asked £or a description of what the computer^uld do (6) interpret -loopit^tg 
problems gaVe a looping program and required a description of what the computer 
would do. Examples of the six'lproblems are given Table 2. 

Results > , The proportion correct response by type of probl^ for each of 
|he treatment 'groups vis given ia Table 3. As can be seen,^ the control gri'up 
performs as well or better on probleiis that are very much I'ike the' material in 
the^l^tructional manual, e.g., genefate-s'tateint and generate-nonloop . How- ' 
.ever, on problems that require moderate amounts of transfer"^— e.g. , generate- 
loop and the shorter interpret problems—the- model group excels. Both groups 
do , poorly on the Very complex interpret-looping programs. The difference in 
the pattern of per/ormance is consistent with earlier result? in other domains" 
in which models enhance transfer performance but not simple retention of pre-r 

♦ 

Sented material. Apparently, the model provided an assimilative context in 
whlph novices ciould relate new technical information in the booklet to a 
familiar Analogy. This learning process resulted in a learning outcome that 
supported some transfer. 



Insert Tables 1, 2 and 3 about het?e 



2.6 Locus of the Effect of Models 

. One problem wltti the above study is that the iodel subjects received more 
Information than the controls. However, assimilation theory (see Introduction) 



•predicts that presenting a model prior bo learning wHl enhance learning be- 
cause it provi^des a meaningful context, bqt' presenting, the model after the text 
will not enhance learning because students will have already -^enci^^d^ the / 
material in a rote way. In furthelr studies (MAYE76a) subjects read th^ same 
BASIC-^like manual, but some subjects were sihown a coiicrete model* of the com- 
puter before' reading while others were shox^i the same model after reading the 
manual. Thus, subjects in the before *group (i.e.; those who received the model 
first) were able to use- the model while encoding the material fn the text, but 
the after gtoup (i.e., receiving the model last) was not. • 

Method . The booklet, model and test were similar to those i^sed in the 
previous experiment. The before group reoeiyed the model, then the-booklet, 

then the test. .The after group received the booklet, then the model, and their 

* J' . 

the test. ' ^ ' ^ 

Results . The proportion of coj^ejct answers by type of problem for the two 
groups Is given in Tabr^ 4. As can be seen, the after group (like the controls 
i^n the previojis study) excels on retentionrlike proJ?lems (i.e., generation- 
statement drid, genei^ation-nonloop) , but the before subjects excel on .problems 
requiring creative; transfer to new situations (i.e., generation-loop, inter- 
pretation's ta temen^^^J^nterpretat ion-nonloop) . Thus, these results provide 
further sppport for the claim that subjects who use a concrete model during 
learning develop learning outcomes fehat support broader transfer. As pre- 
dieted, the locus of the effect is before rather than after instruction. 



Insert Table 4 about here 



2.7 Effects of Models on Recall Performance 

The above studies used transfer tests as a* measure of what is learned 
under different instructional t^hniques'. Another technique -involves asking 
subjects to try to write down air^tl\ey can remember about certain statements. 
In a follow-up study (MAYE^O) subjects read the same' manual and were given the 
model either befbre or ^fter reading as in the previous study. However, as a 
test,, subjects were asked to' recall all the^ could about portions of the manual 
i Meth6d»' The same booklet* and model were used as in the previous experi- 
ments, with some minor modifications. *The before group received the model, 
then th^ mai^ar, then the recall test; the after group received the manual, 
then the model, and then the recall test. 

Results . In order to analyze the recall protocols, the inf/Drmation in the 
manual was broken down into "idea units."/ Each idea unit .expressed one major 
idea or action. There were three kinds of idea units in the manual: 

(1) conceptual idea units related to the internal operation of the computer, 

(2) technical idea units gav.e' examples of code, and (3) forest idea units^ gave 

* 1. 

grammar rules. Table 5 gives examples of each type of idea unit. 

Table 6' shows the average number of correctly recalled Idea units from 
each" category for the two groups. Asf can be Been, the before group recalls 
more conceptual information while the after group recalls more technical and 
format information. This pattern is consistent with the idea that good re- 
tention requires recall of specif ic* code, but good transfer requirefe under- 

i * * ^ * s 

standing of conceptual ideas. Also, Table 6 shows -^at the bef(^re group in- 
cluded more iijtrusions about the model and about other idea unit^ from other 
sections o^ the bookleC, thus suggesting they integrated the infc^rmation 
better. For example, an intrusion is, "An address is a slot in thi memory 
.scoreboard.^" The after* group, however, included more vague summarises and 
connectives which served as "filler." For example, a connective ik "And that's 
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(with very few operations) to a comput^-2 program (with many different state- 
ments Integrated into one large program) . Table 8 lists jthe five different 
kinds of programs used. « ' ' . 

Results. Table. 9 gives the proportion of correct answers by type 
•problem for the two' treatment groups. As can be seen, the control group 
performs as well as the model group^n very simple problems like tho^ in the 
manual, but the model group excels on longer prpb.iems that require creatively 
integrating all of the statements in the booklet. Thus,, as in the studies 'with , 
Basic-like materials, a familiar model serve's to enhance performance on cre- 
ative transfer When it is p'respnted pripr to technical instruction. 



Insert Figure 3 and Tables 7, 8 and 9 about here 



2.9 Abilitv 



The pattern of results described above tended to be strongest for low 
ability Subjects (MAYE75b) where ability is defined in terms of, Mathematics SAT 
. score. For exkmple, for low ability subjects the advance organizer increased 
transfer test' performance (55% correct) as compared to the control group (45% 
correct), but for high ability learners the advance organizer group performed 
more .poorly than the control gtoup (55%. versus 62% correct, respectively),. ^ 
; Apparently, high ability learners already possessed their own useful "models"* 
^for thinking about how. a computer wqirks, but low ability students are more 
Hkeiy to lack useful prerequisite knowledge. 
2>10 Text Organization 

The patteri) of results described above also tended to be strongest when 
material was poorly organized (MAYE78). For example, the Basic-like manual was 
presented either in its Original order or in a random order. In the random 



how READ statements work." Thus, as with the transfer test,' subjects given the 
model before learning show evidence of more integrated and conceptual learning 
of technical information. s 



Insert Tables 5 and 6 about here 
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2,8 Effects of Models on Transfer and Recall Using a Different Language 

Although the above results are consistent and were obtained in a long 
series of studies^ their generality^ is limited by the fact that just one type 
of language was used,' Thus, a follow-up study (MAYESOa) was conducted using 
a file management language based on SEQUEL (GOUL74; REIS 77). The goal of this 
study is to determine whether the resSi^ts from previous studies generalize to a 

new domain. i ■■z'^ 

/ • 

1 

Method . Subjects read a manual t^t presented the file management lan- 

o ' j 

guage. For one group of subjects, the model group, the manual began with. 

\ * - • i r 

discussion of a concrete model and relaited each statement to the model (see 

Figure 3), but no model was given to the control group. The manuals were 
X 

Informationally equivalent. Each page of the booklet presented one of the 
eight stat^mi^nts shown iii Table 7^ alq^tig with examples of how the statement fit 
into Ec program. Figure 3 presents the concrete model that was used: long-term 
memory is represented as a file cabinet; the sorting function is represented as 
an in-baskety out -basket, and save basket; temporary memory is represented as 
an erasable scoreboard; executive control is represented as a list and pointer 
arrow; output is represented as a message pad. The entire model was presented 
on a 2 X 3 foot diagram,^ in order to enhance the learner's ability to visualize 
the system. ^ 

After reading the manual, all subjects took the same 20 item test. 
Problems varied in complexity from generating or interpreting a sort-1 program 
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^rder , presentation order of paragraphs was randomized. For the randomized 
version of the manual, the advance organizer group performed l^etter on a 
transfer test than a control group (41% correct versus 33% correct, respec- 
tlvely); but for the logical version of the manual advance organizer group 
performed but did not outperform the control group (36% versus 44% correct, 
respectively). Apparently, the model Is more useful vhen material Is poorly 
structured because It helps the reader to hold the Information together. 
2.11 Conclusion 

These results provide clear and consistent evidence that a concrete model 
can have a strong effect on the encoding and use of new technical information by 
novices. These results provide empirical support to the claims that allowing 
novices to "sea the works" allows them to encode information in a more coherent 
and useful way (DUB076, DUB078) . When appropriate mod^s are used, the learner 
seems to be able to assimilate each new statement to his, or her Image of the 
computer system.' Thus, one straightf ori/ard implication is: If your goal is to 
produce learners who will not need to use the language creatively, then no model 
is needed. If your goal is to produce learners who will be able to come up with 
creative solutions to novel (for them) problems, then a concrete model early in 
learjolng is quite useful. More research is needed in order to determine the 
specific effercts of concrete models on what is learned, and to determine the 
characteristics of a useful model. 

3.0 DOES STDDEtJT ELABORATION ACTIVITY AID MEANINGFUL LEARNING? 

3.1 Statement of the problem 

* The previous section provided evidence that concrete models may influence 
learning of computer programming l^ecause they prOvide a familiar context for 
assimilating the new material. The second major technique for increasing 
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tlie meanlngfulness of technical Information Is elaboration — encouraging the 
learner to explain the Information In his or her own words and to relate the 
material to other Ideas or concepts. Elaboration techniques may Influence 
meaningful learning because they encourage the activation of existing knowledge 
that is relevant for comprehending the newl;^ presented material, i«e«^ elabor*- 
atlon may af f ect"^ the activation process (see Figure 1) • 
3>2 Putting It in Your Pwn Words 

There is some evidence that asking subjects to put ideas into their own 
words during learning tan enhance the breadth of learning. / For example, Gagne 
& Smith (GAGN62) asked subjects to give a verbal rationalization for (Bach step 
aa they learned to solve a three disc version of the Tower of Hanoi problem 
(EWER32). These subjects took longer to learn than those who did not verbalize; 
however J, they were abl^ to transfer what they had learned to different problems, 
such as a six disc version, much more efficiently (e.g., 3. 8 minutes to solu- 
tion) than the non-verbalizers (e.g., 10.0 minutes to solution). 

More recently, Wlttrock (WITT74), has proposed the idea that "learning is 
a generative process"-^. e., learning occurs when the learner actively generates 
associations between what is presented and what he already has in^ memory. As 
an example, Wlttrock (WITT74) presented a study in which elementary school 
children read a passage and either generated a one-sentence summary for each 
paragraph or did not. Recall by the students who generated summary sentences 
was nearly double that of the control group. Apparently, when students are 
actlvel^r. encouraged to put information in their own words, they are able to 
connect the new information to existing knowledge. 

Elaboration techniques have long been used by experimental psychologists 
to enhance the learning of paired associates (such as HOUSE-CASA) . For example, 
irfien students are asked to actively form mental images or a sentence involving 



word pairs, paired associate recall is greatly enhances (BOWE72, PAIV69). More 

irecently, elaboration techniques have been used in school curricula (DANS78, 

WEIN78>. For example, in studying human physiology, students are asked "How do 

arteries differ from veins?". Several researchers have argued that jptudents 

should be given explicit training in "learning strategies" for actively pro- 

i 

cessing new material (ONEI78) . 

The following is a series of studies that explore the* role of elaboration 

techniques in learning computer programming. The main theme ot this research 

is to determine -how "putting it in your own words" influences the learning of a 

1 

new computer language. 

3.3 Effects of Model Elaboration on Transfer Performance 

The fir|t set of studies (WAYESOa) address the question of whether elab- 
oration activity influences students' ability to engage in problem solving. In 
these studies, subjects learned a new computer programming language and either 
were or were not encouraged to describe what they learned in their own words by 
relating it to a concrete familiar situation. " ^ ' 

Method. Subjects read an instructional manual covering an information 
management lai^ge similar to that described in the previpus section (see • 
Tables 7 and 8). For subjects in the model elaboration group, there was an 
elaboration page after each page in the manual while for subjects in the con- 
trol group there was no elaboration exercise. The elaboration exercises asked 
the subject to describe the newly learned statement in terms of operations 
wlttiin a concrete model of the computer. Table 10 provides a typical exercise. 
Then, all subjects took the same 20 item problem solving test as described in 
the previous section. 



V; . 

Insert Table 10 and 11 about here 
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Results ^ Tbble 11 shows the proportion bqrrect response by type of problem 
2 

for 'the tvo groups.' As can be seen, the control group performs well on simple 
re tentlon«*like problems ^ but the model elal^oration group performs considerably 
better on problems requiring creative transfer. Thus, there 'Is evidence that 
requiring the learners to put technical information in their own words though 
relating the material to a familiar situation, results in broader leamipg 
outcomes. The results are similar to those given in Table 9» and Suggt&st thai: 

model advance organizers and model elaboration have similar e'ffects. 

- - . » 

3>4 Effctcts of Comparative Elaboration on Transfer P^fforaiince 

In the previous study, a concrete 8ituat:ion is presented and the learner is 
asked to relate the new information to It. However, the results are ambiguous 
in the sense that they may be attributed either to elaboration activity per se 
or to the f act^ that additional inf oan&ation; (about the concrete ioodel) was pre- 
sented to the model elaboration group^ The purpose of the present studies were 
to use a kind of elaboriatlon activity that does not add new information (MAYESOa) 

ThuSy a set of studies were conducted in which some subjects were asked to ' 

■ ■ # 

compare newly learned statements in their own words. 

♦ ■ , ■■ • 

Method . Subjects read the same manual about an information management 
language as in the previous study. However, some subjects were g;lven an elab- 
oration page after each page in the booklet (comparative elaboration group) , 
while for other subjects^^tKere was no-elabpration (control group). ^ The elab- 
oration activity asked .subjects to tell how two statements were similar and 
different,, In their own words. Table 12 provides a typical exercise. Then, all 
subjects took the same test as in the previous study. ^ ^ 

^Sult6 > Table 13 shows the proportion of 'correct answers by' type of 
problem for the two groups. As can be seen, the control group excels on 
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retention-like problems but the comparative elaboration groups excels on the 
more complex transfer problems. Thus, there is evidence corresponding to that 
found in the model elaboration studies, that asking learners to put technical 
information in their own words (through making comparisons) results in broader ^ 
^learning which supports transfer. 



Insert Tables 12 and 13 about here 

3,5 Effects of Model and Comparative Elaboration on Recall . • - 

The previous studies suggest that elaboration activity can Influence 
transfer performance. As a further test (MAYESOa) subjects wfere given manuals 
with either no elaboration questions, model elaboration questions, or compara- 
tive elaboration questions. It c^n be predicted that the elaboration subjects 
fifaould. recall more^ information that supports transfer — such as conceptual in- 
formation--while the control group should recall more Information about specific 
8tatement8~-such as technical information. 

Method. As in the previous , study, subjects read a manual explaining the 
information managBment language tha^ contained either no questions (control 
grpup) , model questions (model elaboration group) , or comparative questions 
(comparative elaboration grcfUp). Then, subjects were asked to recall portions 
of the text. ^ 

Results . For purposes of scoring the recall protocols, the text was - 
divided into idea units. Some of the idea units presented information about how 
the conqiuter operated (conceptual idea units) and others emphasized the grammar 
and technical aspects of each statement (technical idea units). Table 14 shows 
the average number of idea units recalled by type for the^ three, groups. As can 
be seen the control , group recalls equal amounts of both types of information. 



but thcT elaboration groups each tend £o emphasize recall of conceptual as com- 
pared to technical information. These results are consistent with the resiults 
of the transfer studies » in that conceptual Information is likely to be needed 
to support transfer. 

Insert Table 14 about here 



3.6 Effects of Note*"taklng on Transfer and Recall Performance 

The foregoing series of studies provides some evidence that elaboration 
techniques influence the breadth of learning. However» the generality of the 
result is * llioited by the fact that just one type of manual was used and Just two 
types of elaboration activity, . In addition ^ previous studies did not control 
for amount of reading time. Thus» an additional series of studies (PEPE78> was 
conducted using a different language (a BASIC-like language) and a different 
elabor^Llon activity (note-taking). 

Method . Subjects watched a 20 minute videotape lecture describing seven 
BASXC-like statements similar to the manual described earlier. Some subjects 
were asked to take notes » by putting the basic information in their own words. 
Other subjects simply viewed the lecture without taking notes. As a test» 
subjects jiTere given problems to solve or asked to recall portions of the lesson. 
Videotape presentations controlled for presentation time in the two groups. 

Results . Table 15 gives the proportion of correct answers on generative 
prohlems (similar to those in the lesson) 'and on interpretation problems (which 
were not in the iMsbn). As can be seen^ for low ability subjects (based on 
Mathematics SAT scores)^ there 4s. a pattern in which note-taking helps perfor- 
mance on transfer but Jiurts performance on the retention- like problems. For 
high ability subjects » note-faking has no effect » presumably because high ability 
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learners already pdssesd. strategies for actively assimilating the new infor- 
mation. 

Table 16 shows Recall :ojE the lecture by type of idea unit for the two 
groups. As can be seen, the note takers recall more conceptual information, but 
there ia no diiEference between the groups in recall of technical information. 
Thus, the results are consistent with the model elaboration and comparatl;ve 
elaboration studies concerndLnyg, the effects of asking subjects to put new tech- 
nical information in their words during learning. 



Insert Tables 15 and 16 about here 



3.7 Conclusion ' 

~ ~l . 

The goal of elaboration is to help ^he learner to be able to describe the 
key concepts in his own \#ords, using his existing knowledge. . Unfortunately/ 
there is no fool-proof way to design useful elaboration activities. Emphasis on 
format or giammatical details and emphasis on errorless verbatim recall of 
statements will not product the desired effects. The learner should be able to 
describe the effects of each program statement in his own words. 
4.0 UNDERSTAMDING COMPUTER PROGRAMMING 

The previous section's have focused on the issue of how to teach novices. 
This section briefly examines' the issue of what to teach. Greeno (GREE76g^has 
argued that instruction for problem solving C^sks should be based on cognitive 
objectives— statements of what the learner should haVe in his or her head at' 
the end of instruction. Two major objectives that are relevant to enhancing a 
novice's understanding of computer programming are: knowledge for understanding 
« stbteoent and knowledge for understanding a program. 
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4.jy;;y|^erjitandlne a Statement 



dbes S^t mean to say that someone "understands" a certain statement? 
In a re^c^nt 'Analysis of BASIC » each statement is described as a "transaction" 
(lfAYE7dc). A "transaction" consists of an action^ an object » and a location in 
the comi^Ker. For example ^ the statement LET X " 5» consists, of the following 
six tr«fchf actions:^ - 

1. '^I^ji^nid ^4^^ number Indicated on the right of the equalfs. j 
(ACTION: Find; fl^^ Number; LOCATION: Program). 

2* Find. tne'^^lBifdaber in the memory space indi'bated on the left of the , 

(ACTION: Find; OBvlECT: Number; LOCATION: Memory). 

■ - , < V- > 

3.^^ Erase th^j:ftixmber in that memory space. (ACTION: Destroy; OBJECT: 

. - v ■', - . 

Writfe the new number in that memory space. (ACTION: Create; OBJECT: 
Number; LOcXxiON: Memoify). 

5. Go on tol^e next statement. (ACTION: Move; OBJECT Pointer; LOCATION: 
Prc/gram) . 

6. Do what it says. (ACTION: Allow; OBJECT: Command; LOCATION: Pro- 
gram). . 

<:Thu8» there general structure for each transaction; you can expect sofie 

action to be carried out on some object in some location in the computer. The 
two techniques cited in previous sections can be applied to teaching a trans- 
action-type analysis of statements. It may be noted that statements with the 
same name may actually cpnsist^of different actions. For example » a "Counter 
Set LET" such as above is different from an "Arithmetic LET" such act^LET X 
■> 5/2. Explicit ngmlng and describing of different' types statements with the 
keyword may become a part of computer Instiructlon. 
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4.2 Understanding a Program , 

What do experts know about compjiter programming that beginners do not 
know? One answer is that experts possess much more information and tha*: the 
ii^ormation is organized mote efficiently,. For example, a review, of research 
on teaching t>eople how to become better problem solvers concludes that Igood 
problem solving requires that the user has domain-specific knowledge: "All 
problem solving is based on knowledge" (GREE80). Similarly, Simon (SIM080) 
estimates* that a person needs 5^,000 chunks of domain-specific information to 
become an expert in some domain. s' ^ ' 

In* a classic ^tudy, subjects were asked to" view briefly presented chess 
? board configurations and then try to reconstruct them (CHAS73)\. Chess masters 
perfbtm^d much better than les^ experienced player^ on reconstructing board 
configurations if the board positions came from actual games; however, the 
advantage was lost when tandom board patterns were presented. This finding 
suggests that experts JLn chess do not necessarily have betster memories, but 
rather they have a repertoire of tbany meatiingful patterns of. board positions. 
They can chuiik sevferal pieces together into one meaningful pattern while a less 
experienced player must try to remember each piece separately. In an analogous 
• study reportfcd by Shnelderman (SimfiSO), experienced and inexperienced pro- 
graaiiners were given programs to study. Experienced programmers were able to 
recall many more lines of code than inexperienced programmers when the program 
was a meaningful running program; however, when the program consisted of random 
lines of code' the two groups perfor^d at similar levels. Apparently, the 
experts were able to chunk' JinQs of code together intd chunks while less ex- 
perienced users were les§ able to form such chunks. 

- #- 

For example, Atwood & Ramsey' (ATW07§) suggest that experienced prograramei/s 
encode a segment such as, 

ERIC , \ 



1 SUM - 0 
DO 1 1 - 1, !r 

SUM - SUM + (I) 
1 CONTINUE ' 



as "CALCULATE THE SUM OF ARRAY X." An experienced programmeir has a "schema" 
for this task ah<) is able to generate a variety of lines of code to accomplish 
it. In order to provide a more precise description of the "schemas" that are 
Involved in understanding programs » Atwood- & Ramsey (ATH080) used a modified' 
version of Klntsch's (KINT74) propositionai analysis. Each stat^ent; in the 
program can be written as a predicate with arguments » and a macrostructure can 
be constructed. Although a detailed description of Atwood & Rautee/'s^ system is 
beyond the scope of this paper » their work is promising in that it miggest:s 
that knowledge can be represented precisely. 

One implication of this work is that it might be possible to explicitly 
teach the major/*chunks*J or "schemas" involved *in computer programming using 
the techniques cited in prievious sections. Explicit naming and teaching of 
basic schemas such as these may become part of computer programming- curricula. 



SUMMARY 



^This paper is concerned with how to make computers and computer pro^ ' 
gra]nid.ng more understandal^^ for novices. Two instructional techniques from 
educational and' cognitive psychology are desgribed — using concrete models to 
represent the computer system^ and encouraging the learner to desetlbe tech- 

y 

nical information in his own words. A review of the effectiveness of these 
techniques revealed that/ under certain conditions, both may enhance the 
learner's understanding as measured by ability to solve transfer problems. 
Finally, two major objectives of computing instruction, were suggested — en- 
hancing « the novice's ability. to understand statements and to understand programs 
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FOOTNOTES 

Transfer problem3 are problems that are different from those given In 
the text, but can be> solved using. Informatlon'in the text. Since 
the text "gave information about how to generate single statements and 
simple programs, these two kinds of problems are not transfer problems. 
Since the text jdid not explicitly^ mention looping^ problems that require 
the generation of a looping program are transfer problems. Similarly, 
since the text did not explicitly deal with interpretation of programs^ 
^ interpretation problems are transfer problems itf/this study. However, 
looping-interpretatioia may require much more transfer thdn the others, 
since it is most different from the^text. 

2. These tables are broken down by problem complexity, with more complex 

r 

problems requiring transfer. The same general pattern Is found for 
both generation and interpretation problems. Table 13 shows data for 
interpretation problems only, in order to avoid unnecessary complexity. 
However, this table cannot be directly compared with Table 11 • 
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Table 1 

Seven Statements Used in BASIC-like Instructional Booklet 



Name 


Example . 


READ 

t 


PI 


READ (Al) 


WRITE 


P2 


WRITE (Al) 


EQUALS 


P3 


> 

Al - 88 


CALCULATE 


P4 


Al » Al + 12 


GOTO 


P6 


GO TO PI 


IF 


P5 


IF (Al - 100) GO TO P9 


STOP 


aP9 


STOP 



r 



I 



Table 2 

Examples of Six Types of Test Problems for a BASIC-like Language 

Geneyatlon-Statement Interpretation-Statement 
« 

Given a number in memory space " A5 » 0 

A5 » write a statement to change 
that number to zero. 



Gen6ration-Nonlo6p 
Given a card with a number 
on it is inputs write a 
program to prints out its 
square. 



Inter pret at ion-Nonloop 
PI READ (Al) 
P2 Al « Al * Al 
P3 WRITE (Al) 
P4 STOP 



Generation-Looping 
Given a pile of data cards 
Is Input, write a program to 
print out. each number and stop 
wfa^ It gets to card with 88 
on it. 



Interpretat Ibn-Looplng 

PI READ (Al) 

P2 IF(A1 - 88) GO T0-P5 

P3 WRITE (Al) 

P4 GO TO PI 

P5 STOP 



Table 3 

Proportion of Correct Answers on Transfer Test by Type of Problem for Model and Control Groups 



Generation 



Irtterpretation 



Statement Nonloop 



Looping 



Statement Nonloop Looping 



Model Group ^ 
Control Group 



.63 
.67 



.37 
.52 



.30 
.12 



.62 
.42" 



.62 
.32 



.09 
.12 



Note. 20 subjects per group; interaction between group and problem type, p < .05. 



Table 4 ' ^ 

Froportlojl of Correct Answers on Transfer Test by Type of Problem for Before and After Groups 



Generation 



Interpretation 



Statement ilonloop > Looping 



Statement Nonloop Looping 



Before 
After " 



.57 ' 
.77 



.50 
.63 



.20 
.13 



.47 
.27 



.63 
.40 



a7 

.17 



Note. 20 subjects per group; Interaction between group and problem type, p < .05. 
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' T^ble 5 

Example of Conceptual , Format/ and Technical Idea Units 

J* 





Idea Unit 


Technical 


READ is one kind of statement. 


Format 


The format is READ ( ) . 


Format 


'An address name goes in t;her parenthesis. 


Conceptual 

♦ . * 


Ah address name \t a space in the computer's memory. 


Conce'ptua 1 
« ft . 


. There. a re 8 memory spaces. 


Technical 


The spaces are called Al» A2 .... 


Technical 


An example is, READ j(A2) . , • - ^ 


Conceptual 
« 


First y the computer checic's the number frpm the top data card. 


jConceptual 


Then, that number is stored in space Kl. < 


Conceptua 1 

% 


The previous number ih A2 is destroyed^ 


" Conceptual 


Then the data card. is sent out of the computer. 


Conceptual 


This reduces the pile of dd(ia card by 1. 


Conceptua I 


Then, go on to the next statements. *' 



• * • ' , r • 

- . ■ 9 ' . • ' 

' Table 6 

Average Number of Jlecalled Idea Units for the Before &nd After Groups^ 

Idea Units . ' Intrusions . 

Technical Format Conceptual ^ Inappropriate Appropriate Model 





(14) 


(12) 


(35) 








Before' 


5.6 


1.9 


6.6 


1;5 


1.3 


3.1 


After 


6.0 


2.9 


• 4.9 , 


2.5 


.8 ' 


.5 



Note.. 30 subjects per group; interaction between group and problem typ^, p < .05. 
Numbers In parentheses indicate total possible. 

A. 

* 



55 

id 



51 



• f> 

r 

Name 
■ ' i 

FROM 
FOR ' 
AND FOR 
OR FOR 
LIST 
COUNT. 
TOTAL 
LET 



Table 7 

Eight Statements Used in File Management Language Booklet 

t 

Example 
/ 

FROM AUTOMOBILE 

FOR WEIGHT IS CALLED 3000 OR MORE 

AND FOR COLOR IS CALLED GREEN 

OR FOR MAKE IS CALLED FORD 

LIST NAME ^ . 

CdUNT 

TOTAL CURRENT VALUE 

LET TOTAL t COUNT BE CALLED AVERAGE 



/ 



.ERIC 



5d 



52 



Table 8 

Examples of Test Problems for a File Management Language 



Sort 1 



List the owners* names for all 
cars weighing 3000'pounds or more. 



FROM AUTOMOBILE 

FOR WEIGHT IS CALLED 3000 OR MORE 
> LIST NAME > . 



Sort 2 

List the owners' names for all late 
moddl green Fords. 



FROM AUTOMOB I LE 

FOR YEAR IS CABLED 1976 OR MORE 
AND FOR COLOR IS CALLED GREEN 
AND FOR MAKE IS CALLED FO {D 
LIST NAME 



Courft 

How many cars are registered in 
Santa Barbara County? 



FROM AUTOMOB I LE 

FOR HOME COUNTY IS CALLED SANTA BARBARA 
COUNT ~" 
LIST COUNT 



Compute 1 

What is the average current value 
of all cards? 



FROM AUTOMOBILE 
COUNT 

TOTAL CURRENT VALUE 

LET TOTAL ^ COUNT BE CALLED AVERAGE 

LIST AVERAGE 



Compute 2 

What percentage of 1977 cars are 
Chevrolets? 



FROM AUTOMOB I LE 

FOR YEAR IS CALLED 1977 

COUNT 

LET THIS BE CALLED COUNT 1 



AND FOR MAKE IS CALLED CHEVROLET 
COUNT 

LET THIS BE CALLED COUNT 2 

LET COUNT 2 v COUNT 1 BE CALLED AVERAGE 

LIST AVERAGE 



erJc 
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. V " °. Table 9 

Proportion of Correct Answers on Transfer Test for Model and Control Groups- 
File Management Language 

Type of Test Problem * 
Sort-1 Sort-2 Count Computer-1 eompute-2 
M^elTGroup .66 .66 .63 . .58 .45 

Control Group .'63 .44 .43 .33 .22 

. / 

Note. 20 subjects per group; group x problem type interaction, p < .07. 



« 

Table 10 

Exampli^of the Model Elaboration Exercise in the Programming Text 

/ 

Model Elaboration j ' ■ 

Consider the following situation^ An office clerk has an in-basket, a save 
basket 9 discard basket , and a sorting area on the desk* The in-basket is 
full of irecords. Each one can be examined individually in the sorting area 
of the desk and then placed In either the same or discard basket. Describe 
the FOR statement In terms of what oper tat ions the clerk would perform using 
tlie la-^b'asket/ discard basket, save basket, and sorting area. 



/ 
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Table 11 ' 

Proportion of Correct Answers on Transfer Test by Type of Problem for 
Model Elaboration and Control Groups 

Type of Test Problem 

Sort-1 Sort-2 Count Compute-1 Computer«-*2 

Model Elaboration Group .65 .58 .64 .64 .45 

Control Group .66 .64 .41 .38 .27 

Note. 20 subjects per group; group x problem type interaction, p < .05. 
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Table 12 

Example of the Comparative Elaboration Exercise in the Programming Text 
Coaparative Elaboration . 
Hp^ is the. FOR command like the FROM command? 
How is the FOR command different than the FROM command? 



Table 13 

Proportion Correct on* Transfer Test for Comparative Elaboration and Control Groups 

r Type of Problem 

Sort-1 Sort-2 Count Compute-1 Compute-2 
Comparative Elaboration .90 .90 1.00 .75 .55 
' Central .90 .90 .65 .65 .25 

IS ■ • • , 

Note, 'Data is for Interpretation problems only. 13 subjects per group, 
group X problem type interaction, p < .05. 



Table 14 " ' 

Average Number of Recalled Idea Units for Model Elaboration. 
Comparative Elaboration and Control Groups 

Type of Idea Units 
Technical * Conceptual 

' ' (19) ^(52) . 

Model Elaboration 5,3 13,9 

Comparative Elaboration 9,4 14.1 

, * s. 
Control 7.5 7.5 

N6te. 20 subjects per group; group x type interaction, p < .05, 
. for low ability subjects. Numbers in parenthesis indicate 
total possible. 



Table 15 

Proportion of Correct Answers on Transfer Test for Notes and No-Notes Groups 



Low Ability Subjects 
Notes Group 
No-Notes Group 



Prol^lem Type 



Generative 

.39 
.49 



Interpretive 

.56 
.33 



High Ability Subjects 
* Notes Group 
No-Notes Group 



.67 
.60 



.62 
..60 



Note. 15 subjects per group; effect of ability, p < .01; 

interaction between group ability^ and problem type, p < .025. 



Tpble 16 " 'W'^ 
. Average Number of Recalled Idea Units for Notes and No-Notes Groups 

Type of Idea Units 

J ^ ^ 

technical Conceptual Intrusions 

(28) (36) 

Notes Group 10.4 7.2 ,3.'9 

No-Uotes Group * 9.4 4.7 2.4 

Note. 20 subjectSyper group; interaction between group and type of 
recall, p < .025. Numbets in parentheses indicate- total 
possible.* ' ^ 



Stimulus- 





SHORT TERM 
MEMORY . 




— > 
















LONG TERM 

f 

MEMORY 


1 ■ 

! 



Response 



Figure 1 , Some Information processing components of meaningful 
learning. Condition a Irs transfer of new Information from outs-lde 
to short term memory. Condition b Is! availability of assimilative 
context in long term memory. Condition c is activation and transfer 
of old knowledge from long term memory to. short term memory. 
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1 



BOARD 



Input Wintkm 



m 



Pn^grom List a Output 
Mnft^ AiTM Pdd 



PI 
P2 
P3 

J 



Figure 2 ^ A concrfet^ model of the computer .for a BASIC-llke 
language* . • . ^ 



' Figure^s2a. Descriptioa for Model 

The figure above represents a simple computer system which you will learn about 
in this experiment. The. computer is made-up of three main parts: (1) INPUT & 
OUTPUT WINDOWS which allow communication between the computer's memory and the 
outside world/ (2) MEMORY SCOREBOARD which stores information in the computer, 
and (3) PROGRAM LIST & POINTER ARROW which tell the computer what to do and 
what order to go itl. Each of these three pirts will now be explained.' 

INPUT & OUTPUT WINDOW , . 

Notice that to the far left X& an input window divided into two parts. A pile 
of computer cards with numbets punched into them can be put in .'the left part of ^ 
the window; as the computer* finishes processing each card it puts the card' on ^ 
the right side of the input window. Thus when the cortputer needs to find tbe 
next data card, it takes the top card on the left side of the input wliidow; 
when it is done with tiie card, it puts it on the right side; 

On the far right is the output window. This is whejre printed' messages (in this 
case, only numbers can be printed) from tfhe compute^* s memory to tKe outside 
world appear. Each line on the printout is a new message (i.e., a new nui;^ber). 

Thus the computer can store in memojcy- a number that is on a card entered through 
the input window or* it can. print out what it has in memory onto a printout at 
the output window. The statements which -put the input and output windows to 
work are READ and WRITE statements, and each will be explained later on. 

MEMORY JSCOREBOARD 

Inside the computer is a large scoreboard called MEMORY. Notice that it is 
divided into eight spaces with room for one score (one nuir^ber) in each space. 



Figure 2a (continued) 



6i» 



AXso notice that each space is labeled with 4 name — Al, A2, A3, A4, A5, A6,' 
A7, A8. These labels or names for each space are called "addresses" and each 
.of the eight addresses always has some number indicated in its space. For 
example,, right now in our figure, Al shows a score of 81, A2 has the number 17. 
etc. \ , 



\ 

\ 



.It is possible to change th6 score in any of the eight spaces; for example, the 
score in box Al- can be changed to 0, and you will learn how to change scores in 
memory later on when we discuss EQUALS statements and CALCULATION statements. 

PROGRAM LIST & POINTER ARROW - . . 

Inside the computer to' the right of the MEMORY is a place to put a list of 
things to do called PROGRAM LIST and an arrow which indicates what st-p in the 
list the computer should work on. 

Notice that .each line in the PROGRAM LIST has a number so that the first line 
is called PI, the second step is P2 and so on. When a program is inserted in 
the step indicator arrow will point to the first line (PI); when the first step 
is finished the arrow will.go to the next step on the list (P2) , and so on down 
the list. .You will learn how to control the order of steps later on when the • 
IF statement, GO TO statement and, STOP statement are discussed. 
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. Figure 3a « Description for Model 

The computer is capable of three main functions f sorting record cards 
int6 sorting bask^its, remembering numbers on its memory scoreborad, and out- 
putlng Ij^formation to the world through its message pad- . 

To understand the soifting function of the computer "you could think of an 
office worker sitting at a» desk with three sorting baskets, a line pointer 
arrow, and file cabinet with many /drawers. Each drawer of the. file cabinet 
contains a different set^of records; the name of the file is Indicated on each 
drawer. If the workar netejls all the records in a particular file all the 
worker needs %o do is 'op^nj that drawer and take out .all the records. To avoid 
onix-ups the clerk can .take out all the, records oflbnly one file at a tim^; if 
the clerk needs to bring records from a tertain file drawer to his desk, first 
all <^ the records froJui all other filds must be put back, in their prope;r <l[rawers. 
Thus, a worker may have all thq records for oivXy on^ file on his desk at a 
l:lmo. These could be placed in the "in "basket" which is on the left side of 
the clerk ^jS desk — it thus contains all of the to-be-processed record cards. 
Waiting for the office clerk to look at them^t i;n t)ie middle of the desk JLs a 
work area with a line pointer arrow; thd tfLerk may place only one card in the 
work area at a time, and the pointer arrow points to, fust one llQe at a time. 
To the right are two more baskets — the "save basket" and the -"discard basket." 
If a record card passes *the clerk's inspection i1: is placed bn top of fehe pile 
of cards in thef "save basket"; but if it fails it is placed jtn the t'op of the 
pile of cards in the "discard basket.^* The procedure the office worker dses is 
to take the top card from the "in basket"*, place' it in^he'wo|:k arek with a 
pointeij arrow aimed rat one line, and based pn inspection of this line tb move 
that card to either the "save" or '"discard bafeket."' The worker continues until 



all of the the records in the. "in basket" have been processes so that >f he "in 
basket" is empty and the "save" and "discard baskets" contain all the records; 
then, sometimes the worker might be asked to take the pile in either the "save" 
of the "discard basket" and put them in the "in basket" for further processing. 

To understand the memoiry function of the computer, think of a memory 
scoreboard. The scoreboard consists of 15 rectangular spaces like a cIa3&room 
blackboard divided into 15 spaces. Each space has a lable such as C0UNT2, and 
each space has on6 number (of any length) in it. The office worker may count 
all the records that have been stored in the SAVE Basket, and' this number cOuld*' 
be stored in one of the spaces on the scoreboard. liJhen a new number is stored 
in a space on the scoreboard, the old number is erased. However, when the 
office worker copies a number* from one of the memory spaces onto-the output pad 
the ntunber is not erased. 

To understand thfe output function qf the computer think of a telephone 
message pad. To communicate with the outside world the computer can write one 
piece of information on each line of the pad. ^ It is fills all the lines on one 
page, itr will ^ust turn to the next page .and begin with the top line. The 
office worker may wri1:e down two kinds of information on the oytput pad: a 
, number may be copied from one of the spaces on the scoreboard onto the pad (but 
this does not alter the number on the scoreboard), or information that is on 
each card ip the Save Basket can be copied onto the output pad. 
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CHAPTER 3 

TASK ANALYSIS, DEVELOPMENT OF EVALUATION INSTRUMENTS, 
AND P'XNOSIS OF BUGS FOR CALCULATOR LANGUAGE 



Note 



.This article has been published as the following citation: 

M^yer,'^R. E., & Bayman, P. Analysis of student's intuitions about the 
^ operation of electronic calculators. Technical Report No. 80-4. 

Santa Barbara: University of California, 1980. 
This article has been subjnitted for publication as the following citation: 
Mayer, R. E. , & Bayman, P. Users-* misconceptions concerning the • 
' * • • operation of Electronic calculators. ' Cognition & Instruction , under 
. review. 

This research was also presented at a professional meeting as the following: 
Mayer, R. E. , & Bayman, P.. Analysis of students* intuitions about the 
operation of electronic calculators. American Educational Research 
Association, Los Angeles, April 13-17, 1981. 
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Abstract 

Xhlrty-three novice dnd 33 expert users predict what number will be in the 
di4[play of a calculator after a sequence of key presses (such as 2 + 3 +) . The 
perlonnance of each subject on 88 problems is formally described as a 13 line 

A 

prbauctlon system. Conditions are key presses (such as + after a number); 
actions are changes in the display or internal registers (such as display the 
evaluation of the ex^f^Sision in the register) . Large individual differences 
arc observed. Differences among ^subjects include when an expression is eval- 
uated and displayed- (e.g. » after a + key» x key, • key, and/or number key), 

whether or not tjhe display is incremented when two operation keys are pressed in 

• t ■ -. ■ - 

sequence (e.g., 2 + + or 2 x x) , whether or not the display is incremented wheij. * 
an equals is pressed after an operatidh (e.g., 2 + ■=), what the order of 
arithmetic will be in a chain (e.g., 2+3x7). Experts are more consistent 
in their performance and tend to be more likely than novices to base answers on 
standard operating systems. Implications for developing a theory of computer 
literacy are discussed. 
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Users* Misconceptions Concerning the Operation of Electronic Calculators 



Within the past decade electronic calculators have become a part of our 
society, including widespread and rapid acceptance In schools (Mullish, 1976). 
3ased on a survey of articles and editorials published within the past few 
.years In Arithmetic Tjeacher an d Mathematics teacher , as well as a policy state- 
ment by the National Council of Teachers of Mathematics (1976), it is clear 
that calculators will, play an important rple in the education^ of our children. 
For example, *the following statements by mathematics educators are typical; 
"All. students will prcTfit from having access to a calculator" (Gawronski & 
Coblentz, 1976). ^ "I propose that we make fullest possible use of calculators 
in all grades of our school" (Hopkins, 1976). "NoTsince the printing pr^ss 
has any invention had such potential for revolunt ionizing education, particu- • 
larly mathematics education" (Rudnick & Krulik, 1976) . 

However,^in- spite of thd^se 'optimistic predictions and endorsements, the 
research community hadrbeen slow in providing information that might be useful 
in this Impendinc^cuJ-ator-curriculura revolution. For example, most experi- 
mental studies to date have compared changes in achievement and/or attitude 
scores for students who use calculators in school versus students who were :iot 
allowed to use calculators fGaslin, 1975; Roberts & Fabrey, 1978; Roberts & 
Glynn, 1979; Schnur & Lang, 1976; Suydam, Note 1). In a recent review of 34 
studies, most of which were not pubMshed in journals, Roberts (1980) observed 
that there was clear evidence that calculators improve computational efficiency 
but no consensus. concerning effects on higher level conceptual achievement orO 
attitudes towards mathematics. Thus, he concludes that "the research literature 
offers no guidance" concerning how to Incorporat i calculators into school 
curricula. 
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The present paper does not attempt to address the question of whether Cal- 
culators influence changes in mathematics achievement and attitude scores. i 
Rather, this paper is^ based on the idea that since calculators will become a 

0 

part of everyday life for students, it is* impottant to know how people come to 

■if 

I* • 

understand and interact with calculators. The calculator represents a student's 
first introduction to a computer, to a computer language, and to computer lit- 
fracy In general. Pressing a key is analogous to a computer command. In spite 
of tremendous breakthroughs ih development of improved calculaltor hardware for 
the mass market there has been comparatively little work on what Shneiderman 
(1980) calls "software psychology". That is to say, we know very little about 
how people understand calculators, what types of instruction will help people 
become creitive^users of calculators, why some people seem to not use them very 
\iell, or how to design operating systems that make psychological sense. 

Since calculator useage seems so simple and since even childj^en can teach 
themselves to use a calculator in a shprt time, some educators have suggested 
that explicit instruction or concern about users' understanding of calculators 
is not needed (Bell, 1976). This might be correct if one's goal is simply to 
have st^u.dents use the calculator as a "blatk box" that gives answers for mundane 
computation. However, when the goal i3 to promote productive problem solvers, 
there is reason to believe that the student's understanding of how the calcu- 
lator operates is Important. For example, Scandura, Ldwprre, V^eski 4f Scandura 
n(i976) found that some students who are left to teach themaelves develop bizzare 
intuitions; for example, some subjects concluded that the plus (+) and equals 
(«) key did nothing since they caused no visible change in the display. On the 
other hand, Meyer (1980) found that by letting f ourth-gradera explqre the 
functj^ons of the operation keys some of them discovered that pushing the same 
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operation key more than once would cause the calculators to repeat the process 
with the number last entered. Leaving the understanding of the basic functions 
of the calculator to chance /night create as many students with incorrect in- 
tuitions as correct ones. Thus, although two users may be able to use the 
calculator to solve basic computational problems, there may be ]^rge individual 
differences in the way users understand and interact with calculator^. 

Elucidation of individual differences in students' intuitions about the 

• . . *^ 

operation of calculators (i.e\, students' conception of what goes on inside the 
•H>lack box" when a key is pressed) is the logical first step in building a 
theory of computer literacy, and is the goal of the present study. In par^- 
ticular, this study' addresses two related issues. * ' - 

(1) What knowledge do people have about how calculators work? Since most 
users are "self-taught" and seem to ^be able to use their calculators, an im- 
portant issue concerns what they have learned. Since some intuitions may lead 
to more creative use of calculators and to better transfer to computer languages 
(such as programmable calculators or BASIC) it would be useful to be able to 
describe exactly what , people' s intuitions are* Recent research on the cognitive 
analysis of computational skill suggests that two children with the* same ap- 
parent performance! may be using entirely different conceptions of computation. 
Wot example, Groen & Parkman (1972) have developed cognitive models of addition, 
and Woods, Resnick & Groen (1975) have developed models of subtraction. More 
recently. Brown & Burton (1975) have developed a BUGGY program which serves to 
diagnose problems in a child's arithmetic procedures by developing a formal 
description of the algorithm that the child is using. Successful application of 
cognitive analysis tools to the problem of describing computational skill en- 
courages the idea that similar techniques can be used to formalize how students 
understand ^calculator logic. 
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(2) What are the differences In Intuitions among Individuals? For ex- 
ample, do "exgerts" have different intuitions than "novices?" In retSent studies,' 
Larkln (1979) and Simon & Simon (1978) have been able to formalize differences 
between what experts and novices know about solving physics problems. Larkin 
(1979) has beeA able to represent the differences in terms of differences in the 
organization' and size of productions in a production system. Similar tech- 
niques may be applied to representing differences between experts and novices in 
the domain of calculator lise. ^ 

STUDIES 1 AND 2 , . 

The purpose of study 1 was to determine the knowledge that ordinary users 
have concerning th^ operation ''of hand-held calculators. In particular, the goal 
was to formally describe each subject's conception of the, calculator's dperating 
system. The purpose of study 2 was to determine whether the formal descriptions 
developed in study 1 would also describe the conceptions of people who were more 
knowledgeable about operating systems. For purposes of this paper, subjects in 
study 1 are labeled' "novices" and subjects in study 2 are labeled "experts". 

Method 

Subjects 

The subjects in study 1 were ^6 college undergraduates recruited from the _ 
subject pools at the University of Pittsburgh and the University of California, 
Santa Barbara.''" All subjects participated in order to fulfill a requirement for 
their introductory psychology courses. Subjects in study 1 had no previous 
experience with computer programming nor wl^^lK the concept of operating systems.* 
Thirteen of the 46 Subjects in study 1 produced inconsistent performance, so 
only data for the remaining 33 subjects was used for the analyses. Of these 33 
subjects there were 16 females and 17 males, and 26 of the subjects owned a 
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. calculator. The mean ^ge was 18 years, the mean GPA wa«'3.0, the mean SAT- 
Quanti^tive score" was 547, the mean SAT-Verbal score was 491, " 

The subjects in study 2 were 35 college undergraduates recruited from a 
^ course in computer programming at the University of California, Santa Barbara,^ 
^ All subjects had taken at least one- previous course in computer programming and 
were currently in a course that included study iyl^/operating systems, .Of thes§ 
35 subjects, one gave inconsistent performance and one failed to^follow direc- 
tions. The reinaining 33 subjects were retainied for analysis in this stady, 
. There were 13 females and 20r males, and 32 of the subjects owned a calculator. 
The mean age was 21 years, the mean GPA was 2,9, the mean SAT-Quantitative score 
was 669, and the mean SAT-Verbal score was 552. 

The main difference between subjects in study 1 (novices) and study 2 
(experts) is theit all of the experts had formal instruction in computer pro- 
gramming and some introductibn to operating systems while none of the novices 
did; the experts were older, t(60) « 2.66, £ < .01; and the experts had higher 
SAT-Quantitative scores, t(44) « 4.67, £ < .001. Tljus, while the main focus 
was on comparing "liberal-arts" students who had no formal programming 
experience to "engineering" students who had formal training in programming, 
any compatisons between the subjects in the two studies must be*made in light 
of other differences such as age and SAT scores. 
Materials 

The materials in study 1 and study 2 were essentially ;Ldentical. Materials 
consisted of a questionnaire, an instruction sheet, and two two-page problem 
sets. 

The questionnaire was an 8*5 x 11 inch sheet of paper which asked the 
^ subject to indicate his or her age, sex, GPA, SAT scores, year in school, major 
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in schools experience with computer programming, experience with calculators,^^ 
-and lirevious mathematics courses. In 'particular, subjects were asked to in- ^ 
dicate whether they owned, or regularly used a particular calculator, and if so, . 
to give the, name of the modeK ^ In a^ition, subjects were .asked to che^ck a box 
correspjo^ding to ^he average number of -^minutes per week they used a calculator — 
less than 10, 10 td^ 30, 30 to 60, more than 60, 

The instructions for the problems were typed onto an. 8*$ x 11 inch sheet of 
paper. Instructions described a typical Calculator and the task* 

Each of the two problem sets di^nsisted of 44 problems typei<onto two 5^ 

X 11 inch sheets of paper with one problem per double-spaced line. Each problem. 

presented a series of key strokes and'proyided a blank space fpr the subject to 

indicate what number would be in the display. . Each problem contained frdm one 

to seven key strokes and each key stroke was either a single digit (2, 3 or 7) » 

a plus key (+), a multiply key (x), or an equals key (=)• ^ The two problem sets 

(Set A and Set B) provided for a .reliability check since each problem in Set A 

corresponded to a problem df the same form in Set B, and both sets p,resented the 

corresponding problems in the lame order. However, the specific digits used in 

corresponding problem were different, for example, problems ^+3+7 or 2+3x or 

2+»+»+- in set A corresponded to 7+3+2 or 7+3x ot 7+=+=«+«, respectively, in Set 

B. ' The' complete list of Set A'problems is 'given in the left side of Table 1. 

♦ 

Procedure * ' * 

The^ procedures were essentially identical in study 1 and study 2 except 
that subjects were run individually in study 1 and were run as. a group in 
study 2 . ^ , 

First, each. subject filled out the questionnaire. Then the instructions 
were presented. Subjects were asked "to suppose that they had just been given a 
new standard four-function calculator that worked efficiently, and to suppose 
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that they j?ould be using this calculator throughout the session. They were told 
that for each problem, their job was to predict what number would be In the 
display of the calculator after the series of key presses ^assuming the calcu- 
lator was cleared at the beginning of the problem) . Then, one of the problem 
sets was randomly selected and given to "the subject; when the subject finished 
the first set, ^the other set was given. Subjects were asked to put' their 
answers in the space next to each problem and there was no time limit. 

* Results, and Discussion 

Scoring , 

The data for each subject in each study consisted of a number (i.e., the 
subject's answer) for eacfi of the 88 problems. 
Reliability of Performance 

( 

Since two forms of the same 44-pro6lem test were administered to each 
subject, it was possible to determine the reliability of each subject's per- 
formance. For each of the possible answers to the 44 problems in set A, 
corresponding answers, were generated for set B. For example, if a subject gave 
12 as the answer for 2+3+7 in set ,A, the corresponding answer fot 7+3+2 in 
set B would also be 12; if a subject gave 7 as the answer for the above prob- 
lem in set A, the corresponding answer in set B would be 2. Similarly, if a 
subject gave 2 as the answer for 2+=+=+= in set A, the corresponding answer for 
7+«+«+= in set B is 7; if a subject gave 16 for the above problem in set A, the 
corresponding answer for set B is 56. Reliability scpring was conducted by 
matching each of the 44 problems in set A with its corresponding problem in set 

B; if the answers did nof correspond, subjects were given a point, 
* » 

Thirty-three of the 46 subjects in study 1 displayed six or less (i'.e., * 

less than 14Z) non-matching scores between set A and set B. Data for the 13 
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subjects who displayed more than six non-matches (i.e«, over 14% unreliable 
answers) were not. analyzed further • The mean non-matching score for all 46 
subjects In study 1 was 5.3 or 12%; the mean non-matching score for the 33 
selected subjects in study 1 was 2.6 or 6%. 

In study 2, one of the 35 subjects gave more than six non-matching answers 
between set A and set B« Data for this subject as well as for one subject who 
failed to follow directions were not included in subsequent analyses. The mean; 
number of non-ioatching answers for the 33 selected sulS^ects in study 2 was .8 or 
2%. 

One question that may be raised at this point is whether the experts and 
novices differ with Respect to the reliability of their performances. The 
proportion of unreliable novices (13 out of 46) was significantly higher than 
the proportion of unreliable experts (1 out of 35) as determined by a chi- 
^quare test, " 7.28, df « 1, £ F .01. In addition, for the 33 selected 
subjects in each study, the novices produced significantly more unreliable 
answers than the experts as determined by a t-test, Jt(64) « 10.59, £ < .001. 
Thus, as might be expected, experts were more consistent in the way they an- 
swered problems than were the novice^. 

All subsequent analyses are based on answers to set A for the 33 subjects 

t 

3 

in each group. 

Performance of Subjects Compared to Performance of Calculators 

In this section we address th^ question of which calculator, most closely 
fits the ansvyers jgiyfen by the subjects. Of the 33 subjects in study 1,' 17 owned 
simple Texas Instruments (TI) models, three owned Sharp models, one owned a 
Rockw^^ mpdel, one owned a Hewlett-Packard HP-21, and eleven either did not 

own a calculator or could not remember what kind they owned. 

I ■ 
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Since TI, Sharp and Rockwell were the relevant models^ owned by subjects in 
study 1, answers to each of the 44 problems in set A were generated using each 
of the three brands of calculator. Interestingly, while most of the calculatots 
gave identical answers for most problems, there were different answers produced 
by at least two of the calculators on 20 of the 44 problems. 

A difference score was computed for each subject for each of the three 
calculator brands. The difference score was based on tallying the number of 
times th^t the subject's answer was not identical to the calculator's answer for 
the 44 .problems in set A. Mean difference scores in study 1 were 8.8 (20%) for 
TI, 20.8 (47%) for Rockwell, and 14.9 (34%) for Sharp. A one-way analysis 
of variance ris conducted on the difference scores with brand of calculator as a 
within subjects factor. The ANOVA revedled that the difference scores listed 
above were significantly different from one another, F (2,64) = 42.6, £ < .001. 
Supi>lementary Newman-Keuls tests indicated that the score for TI was signifi- 
cantly .better than for Rockwell, but no other differences were significant 
(£ < .05). • . 

Of the 33 subjects in study 2, 23 ownca Texas Instruments (TI) models, two 
owned Sharp models, three owned Casio models, three owned Hewlett-Packard, and 
two either did not own a calculator or could not remember what kind they owned. 
Thus, as with novices the. most frequently owned calculator was Tip However, 11 
of the .33 novices did not own or could not remember which calculator they 
owned, while only 2- of the 33 experts fell into this category. According to a 
chi-square test, this difference between proportions is significant, 
« 6.13, df = 1, £ < .025. 

The mean difference scores in sti/i&y 2 were 9.0 (20%) for TI, 18.2 (41%) 
for Rockwell, and 13.7 (31%) for Sharp. A one-way analysis of variance was 
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conducte^ on the difference scores with brand of calculator as a within subjects 
factor. The ANOVA revealed that the difference * scores listed above were sig- 
nificantly different from one another, F (2*,64) = 23.74, £ ,< .001, Supple- 
mentary Newman-Kuels tests revealed that tl)^ score for.TI was significantly 

i 

better than for Rockwell or fbr Sharp, and that the score for Sharp was sigtvifi- 

caritly better than for Rockvei;L (£ < .OS), 

Thus, both in study 1 and in study 2, there is evidence that TVs operating 

system most closely matches the intuitions of subjects for the 44 problems in 

the test, la comparing experts and novices, there is no evidence of any dif- 

ferences in which calculator gives the best fit; t-tests revealed no differences 

between ex:^>erts and novices with respect to their scores for TI, _t(64) < 1, for 

Rockp^ll, £(64) « 1.58, or for Sharp, t^(64) = 1.27, . y 

Performance of Subjects By Type of Calculator They Own . ' 

< 

The previous section suggested that subjects, in our sample generated per- 
f ormance that more closely matched the performance of a calculator than the 
other calculators we tested. However , since the TI is the. brand of calculator 
that most subjects in our sample* owned, the abov^e results may.be mainljr due to 
experience with TI calculators.^ In order to test this idea, subject^ in study 1 
were divided into two groups: those who. owned a TI calculator (n = 17) and 
those who do not (n « 16). ^ ' 

The mean difference scores for the Tl-owners in study 1 were :^ TI = 8.0, 
Rockwell « 20.0, and S|hprp = 14.1; the mean difference scores for the non-TI 
owners were: TI « 9.8, Rockwell * 21.6, Sharp = 15.8. As can be seen', for both. 
TI~owners and uon Tl-owners, the difference scores are lea&t for TI. .An anal- 
ysis of variance was conducted on the difference-score data with group as a 
between subjects factor and type of calculator as a within subjects factor.. 
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There was a significant difference between the calculators In how well thfey 

matched the performance of all subjects, F/(2,60) = 51\16, £ <^001, ;but there 

was no group x calculator interaction, £ (4,60) < 1. "J^hus, 'there was no 

* • ■ ' « * ^- ■ ■ 

evidence that the non-TI owners^were different f rom TI owners with res'pect to 

their performance being )>est fit by a TI calculator. A separate one-way ANOVA 

was conducted on the data for the non-owners with type 'of calculatgr. as a 

within- subjects factor. The differences in difference score's were ' 

significant, F (2,30) « 37.57, p F .001; subsequent Newman-Kuels tests" showed 

that the score for the TI were significantly better than fot Sha.rp or Rockwell^ 

among non-TI owners <.05). ' 

As an additional analysis, the performance of each subject was labeled as 

Tl-like, Rdckwell-like or Sharp-like, based on which of the three calculators 

produced the lowest difference score f^^r the sublet. Of the 33 subjects in 

' / . - . ^ 

study 1> 29 weire^ classified as Tl-like and 4^ere better ^matched with other 
brands. For the Tl-owners, IS were classified us Tl-like and 2 were best fit by 
other brands i for the non-TI owners, -14 were classified as Tl^lijce and 2 were 
best fit by other brands. A Fisher's Exact test revealed tl[iat ther^ . was no 
evidence of any d}.fferences among the two groups (TI owners versus non-owheiT'S) 
in the proportion of TJ-like subjects. 

[ . 

The performance of the experts in study 2, like the novices in gtudy 1,' 
most closely matches the performance of the TI, but this may be due to the fact * 
that TI is the most prevelant caletilator ui^^d among ,tiie. experts. As in study 1, 
this idea was tested by dividing the experts into those who owned TI calcu- 
lator (n « 22) ,and those who did not (n 11). The mean differenp^ scfcres for 
the^ Tl-owners were: TI = 7.6, Rockwell = 34.4, Sharp - 13.2; th^e main 
difference scores for the non-TI owners were: TI = 11.8, Rockwell = 17.4, Sharp 
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« " An analysis of variance was conducted on the difference score. data' 

with bwnershlp group as a between subjects factor and type of calculator as a 

wp:hin subjects factor. As expected^ there was a significant difference among 

the calculators in how well they matched the performance of all subjects, JF * 

(2,60) « 19.21, £ <.01; however, as in study 1, there was no Interaction 

between group and calculator, (4,60) < 1. Thus, as in study 1', there was no 

evidence that non- TI owners were different from TI owners with respect to 

their performance being best fit by/^a TI calculator^ 

As an additional analysis, the performance of each subje.ct JLn study 2 was 

labeled as Tl-like, Rockwell-like, or Sharp-like, based on which of the Chree 

calculators produced the lowest difference scores' for each subject. Of the 33 

subjects, 2^6 were classified a?? Tl-like and 7 were better matched by' other 

brands. For Tl-owners, ,18 were classified as T?Wlike and 4 were best fit by 

other brands; for the noh-TI owners, 8 were classified as Tl-like and 3 were 

• ^ 
best fit by other brands.^ A Fisher's Exact ^test revealed that there was no 

evidence of any differences among the two groups (TI owners versus non-pwners) 

in the proportion of Tl-like subjects. ' 

This section helps clarify the earlier finding that subjects* performance 
is'^pst clbsely matched by TJ's operating system. Since this finding seems to 
"T)e pjf^sent for both Tl-owners and non-owners it may be attributed to the "in- 
tuitive appeal" of the TI opetating system rather than to users having more 
experience with TI products. However, it shpuld be pointed out that the corres- 
pondence between the calculator's answers and the subjects' answers are far from 
perfect, even when we choose the best fitting calculator. . 
Performance of Subjects by Amount of Experience With Calculators - ^ 

This section explores the issue of whether subjects who differ with respect 
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to how much^ttey use a calculator also differ with respect to Intuiftlons about 
the pperating .system underlying the calculator* In order to address this issue, 
subje^cts In study 1 were divided into two groups based on their reported weekly 
use of calculators: low expierience~les^ than 10 minutes per week (n = 17), and 
moderatte experience*^-10 or more minutes per week (n « 16) . 

The mean difference scores for study 1 on each of the three calculators 
brands was TI « 10.2, Rockwell » 22.2, and Sharp =16.5 for the low experience 
.group, *and TI * 7.*4, RockwerM « 19.3, and Sharp = 13.3 for the moderate ex- 
perience, group. As can be seen, TI. produces the lowest score (i.e., best fit) 
for subjects' performance/ in both groups. An analysis of variance was conducted 
on th^ difference score data wi1;h experience (low vs. moderate) , as a between 
subjects factor and calculator brand as a within subjects factor. Aso^ln previous 
^analyses', there was a significant overall difference among the calculators in 
-how closely they fit* the intuitions of the subjects, F (2,62) = 40.62, £ <*.001, 
and there was no interaction between^. group and calculator, F (2,62) < 1. Thus, 
there was no evidence in study 1 thAt the superior fit of the TI calculator was 
influenced by how much experience a subject had. 

As in the previous section, each subject was classified as being either TI- 
like» Rockwell-like, or Sharp-like based on which calculator produced tlxes least 
number of differences with the subject's actual performance. For the low 
experience group in study 1, 15 subjects were best fit by JI and 2 Were best fit 
by another calculator;^ for the, moderate experience group 14 were best^fit by TI 
and 2 were best fit by" another calculator. A Fisher ' s. Exact test showed that 
there were no significant differences between low experience and moderate 
experience groups with respect- to the proportion of TI-liHe subjects. Thus, 
there Is no ^evidence in ,this analysis that amount, qf experience with ca^lcula^tors 
influence the subjects '^intuitions about the operating systems of calculators. 
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Corresponding analyses were conducted on the data 
for the experts "iti study 2. The mean difference scores for study 2 were:^ TI « 
9.5, Rockwell « 18.1, and Sharp » 14.4 for^the low experience group, and TI » 8.4* 
Rockwell « 18.3, and Sharp « 13.5 for the moderate experience group. An ANOVA 
revealed the expected overall effect, F (2,52) = 18.92, £ < .001; but there was ' 
no Interaction between experience group and calculator,, £ (2,62) < 1, and thus 
no evidence that the superior fit of the TI calculator was influenced by how 
much experience a subject had with calculator^. For the low experience group 
in study 2, 6 of the 8 subjects were best fit by TI while for the moderate 

experience group, 21 of the 25 subjects were best fit by TI. A Fisher's Exact 

* - ' 

test showed there was no significant . difference with respect to the proportion 
of Tl-like subjects among low and moderate experience subjects in study 2. 
Thus, for both experts and novices, there is no evidence that experience with 
calculators influences the subj^ects' intuitions about the operating system of 
calculators. ' ^ " >, . 

Inter-Subject Consistency in Performance ^ 

The foregoing analyses indicates 'that subjects* performance was closest to 
that of a TI calculator, and that this pattern was not influenced by whether 
subjects actually owned^ a TI calculator ilor whether they, had experience with 
using a calculator. However, the foregoing analyses also made clear that*' Ob- 
jects' performance could not be adequately described as corresponding to one 
particular calculator's performance, since the best fitting calculator predicted 
only about 80Z of the answers. 1]^ this section, we explore the question of hvow 
similar or different the subjects* answers were from subject to subject. 
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Table 1 gives a list of the 44 problems In set A as well as the answers 
given by subjects In study 1 and study 2. Each answer that was given by any 
subject Is listed (In parentheses) along with the number of subjects who gave 
that answer In study 1 and In study 2. As can be seen, for each question there 
Is an answer that occurs most often (I.e. , the modal answer) and there may be 
one or more other answers given by some subjects (I.e., alternative answers). 
The percentage of subjects' answers that dre modal answers. I.e., answers that 
correspond to the most common answer for each question is 83% for study 1 and 
81Z for study 2. A*t-test revealed' that the experts and novices do not differ 
with respect to the percentage of modal anisfwers, t(64) < 1. Kolmogorov- 
Smimov's One-Sample tests based on the data for both studies together 
indicated that the following problems produce significant (^ .05) number of 
non-modal answers: 4, 13, 15, 17, 19, 21, 23, 24, 25, 27, 28, 29 and 3f^- 
through 44. Thus, the data in Table 1 en^iourages the conclusion that there are 
substantial individual differences among subjects in their intuitions about 
calculators. In order to more closely examine and describe these differences, 
all subsequent analyses will involve descriptions of single subjects rather 
than group data. 
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Analysis of Performance of Individual Subjects on Simple Problems 

The goal of the analysis in this section is to provide a formal description 
of the li^ovledge that each subject has concerning how the calculator solves 
simple problems. Thus, for each subject, a model was developed which could 
generate the subject's answers on simple test problems. This section describes 
the data source, the format of the models, hpw the data for a subject were fit by 
a particular model. . « 
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Data Source . This analysis was based on the data for the 33 novices and 
the 33 experts who gave consistent answers (i^e., high reliability between the 
two corresponding question sets) . In order tp'^provlde an intensive analysis of 
the performance of each subject in the sample, this analysis focused on only the 
^mple problems. ' Slmple'problems are defined as those which contain symbols for 
number and/or plus and/or equals but which do not' contain any multiplication 
symbols. Eighteen of the 44 problems in set A fit this description; these are 
problems 1 through 10 and 27 through 34 in Table 1. Thus, for each of the 33 
novices and 33 experts, the data source was a list of 18* answers for the 18 
simple problems. . 

Development of Models ^ The goal of the present analysis was to develop 
.simple production system models that would generate the performance of eaqh 
subject on the 18 simple problems. Because of its efficiency and apparent 
relevance for the present task, a production system was used to represent th^ 
knowledge of each subject. A production system contains a list of productions 
with each production consisting of a condition and a corresponding action. 
Conditions, actions* and productions for. the current problem are described in 
this section. - ' ^ 

*jChe relevant conditions ¥or the present analysis are related to having 
pressed one of the^ k^eys on the calculator keyboard. The key relevant to the 
simple problems are the ten number keys (0, 1, 2, 3, 4, 5, 6, 7, 8, 9), the 
addition operation key (+) and the equals key («) . Table 2 shows nine con- 
ditions 'that' could ^xlst; three are not relevant to the simple problems since 
these conditions are never found in the problems. The other six conditions 
contain att exhaustive list of the conditions present in thi^ simple problems. At 
flral^ blush, it 6ilght seem that Table 2 gives many redundant conditions and that 
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" e slmpller scheme is to deal with only three .basic conditions — pressing a 
number (#) , pressing a'plus (+), or pressing an equals (=^). However, the 
'performance and comments ^ subjects suggest that a key press means something 
different to some people depending on the immediately «preceeding key\press. 
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The actions for each production consist of events that take place inside 
the calculator. Following a system developed to represent the "internal" 
attions in ccjwputer languages (Mayer, 1979), each of the calculator actions- can 
be i^epresenred as a triplet: some operation is applied to some object at some 
Ideation in tlui calculator. The operations consist of ^he following: 

(1) Create — A number or expression is placed in the display or 

register, e.g., when you press a number key that number 
appears in the 'display. ' 

(2) Destroy — A number or expression is erased from the display or , 

' register, e.g., when you press the equals key the pre- 

vious number in the display is erased (and replaced 
with a new one) . 

(3) Evaluation An expression (from the register) is converted into a 

single number, e.&., the evaluation of 3 + 2 is 5. 
(For the current example, evaluation of a number ot a 
V number followed by an operation is the number, e.g., 

' evaluation of 3 is 3, evaluation of 2 + is 2). 
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Tlife obj^ects consist of: ' ^ 

(1) Numbers — A number is any single or multiple digit sequence 

such as 2, 14, 156, etc, 

(2) Operation — An operation is a mathematical symbol for some arith- 

metic computation such as addition (+) , or multiplication 
(x). 

(3) Expression — An expression is a sequence consisting of numbers and 

operations such as, 2+3 or 2+ or 2+3x4, 

The t^cations* consist of: 

,(1) Diaplay — The^-eKternal display ,in ,a Calculator nq^mally consists 

of at least eight places, where each place can hold, 
one digit, 

if 

(2) Register An internal register is inside the calculator and 

consists of subregisters for individual numbers atid 
for operations. Expressions are held in the order of 
input, with the first number on the left, followed by 
the operation, followed by the next number. 
Table 3 shows some typical actions that may occur for the simple problems. 
It should be noted that the 12 actioTi!fe listed in the table actually refer to ' 
groups of single actions. For example, D « # consists of two single actions: 
erasing the old number from the display and replacing it with a new number. 
Also, it should be noted that the. first five actions refer only to the display, - 
and have nd effect on changing the register; tne other actions refer only to 
the register and have no effect on changing the display. 
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Since Table 2 glVes a list of all relevant conditions and Table 3 gives -a 
list of relevant actions, it is now possible to describe any production as one ' 
of the conditions coupled with one of the actions. For examp]^e, the production, 

P5. If # after + Then D =■ # and R = R + # 

* * ■ 

means that when the number key is pressed after the plus key the result is that' 
the old number in the display is replaced with the new number, and the old 
expression in the register is retained but the new number is added to the 
right. Thus, if the sequence had been 5 + 2 + and now a 3 is pressed, the 
display will contain a 3 (before there was a 2) and the register will contain 
the expression 5+2+3 (before there was 5 + 2 +) . . . 

Fitting a model to each subject . The foregoing sections described that Iftie 
data source for each subjefct was' the 18 answers to the targe,t questions and that 
models are based on (alternative) actions associated with each of the six . 
conditions. Thus, for the data of each subject the goal is to develop a pro- 
duction system which consists of six prodixctions. . ' 
^ The task was made somewhat easier by the fact that there are groups of 
subjects who gave i4i5ntical answers for the simple .problems. Based on their 
answ^s to the 18 simple problems, subjects could be grouped in one of six 
distinct categories where subjcocs in each. of the first five groups produced 
identical answers with one another. Group 1 contained 8 novices and 11 experts, 
all of whom gave answers that were identical to answers produced by infexpensive 
TI inodels%^ Group 2 contained 10 novices and 7 .experts, all of whom gave answers 
that were identical to Group 1 except fo/ situations in which a plus key was 
pressed. Group .3 contained 5 novices and 2 experts, all of whom gave answers 
that were identical to Group 1 except for situations in which a number key was 
presSwd.' Group^A contained 2 novices and 4 experts, all of whom gave answers 
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that were similar to those produced by an J.nexpenslve Rockwell model. Group 5 
contained one novice and 3 experts-, all of^whom showed a mixture of acting lljce V 
Group 1 and Group A. Tfinally*, there were 7 novices and h experts who gave ^ 
idi(/syncratic patterns of answers for the simple problems. Some of the subjects 
In this ciitegory were vary similar td^^^e^Qf the above groups except for one or 
two minor deviations, while others seemed to have^^liighly unique answers. In allv 
cases, however, the answers wer^ internally consistent within subjects as in- 
dicated^ by high correspondence between answers in set A and set B. The answers 
for each of the common categories on eat h "of the 18 simple problems ^re given ,in 



\ Table 4 . 
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Models were^ generated for each of the common' categories ot answers and for 

* » - *3 ' ■ ' 

each subject in the miscellaneous category. The productfion systems for each of 
the five conMon categories ar6 given in Tables 5 through 9, respectively.- 



Insert Tables 5 through 9 about here 



Groups 1, 2, and 3 All behave similarly but differ:, with respect to when an 
expression is evaluated and displayed. For example, CQjisider the sequence of 
key strokes: 2 >«- 3 + 4 « . 

According to model 1, subjects think that the calculator evaluates 'ex-, 
pressions only when a plus key (+) is pressed after a number or when an equals 
'key (») is presded^ Thus, in the above example, the ^number in the display after 
each of the six key ^strokes will be, 2, 2, 3, 5, 4, 9. The. 3 does not get added 
to the 2 until, a plus (or an equals) key is pressed; and the subtotal 5 does not 

gat added ^o A untir an equals (or plus) key is pressed* 

• . ■"■ •• ' ' . ^' 
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AccordiAg to model 2, su^ects think that the calculator evaluates ex- 
pressions only when an equals (=) key is pressed. Thus, in the above example, 
the numbers In the^'display after each of the six key strokes will be 2, 2, 3, 3, 
4, 9.^ The entire expression is held in the register until an equals key is 
pr^ssed.^ ; ' V 

According to model 3, subjects think that the calculator evaluates ex-' * 
pressions as soon as a number key (//) is pressed* Thus, in the above example, 
the numbfera in the display after each rfespec'tive key stroke will be: 2, 2, 5, 5, 
9, 9. The 3 gets added to the 2 as soon as the 3 is pressed; the- 4 gets added ^ 
to the subtotal 5 as soon as the 4 is pressed. 

In summary model 2 involves delayed evaluation (i.e. ,*->no thing gets eval- 
uated until an equals is pressed), model 3 invclves immediate evaluation (i.e., 
expressions are^evaluated as soon as possible), and model T involves a com- 
promise between th^ two extremes (i.e., expressions arie evaluated after a plus 
but not after a dumber key is pressed). Another way to describe the differences 
araong the first three groups is to say that group 1 treats a plus key like an 
equals, group 3 treats both a number key ^d a plus key like an equals, group 2 
treats neither like an equals. 

' The 1 ourth and fifth groups differ from the first three groups with respect 
to how to deal with the plus key. The fourth %tq\x^ behaves as if the calculator 
has an automatic constant — evaluation of an expression for a plus or an equals 
Involves adding the number in the display to the number in the register. This 
mode of evaluation is called "incrementing display" in the tables. The fifth 
group gives an' "incrementing display" only when two plu^ keys are pressed in 
sequence. Both groups are like group 1 in that they display the evaluated 
version of the expression when a plus key or an equals key is pressed but not 
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when a number key is pressed. The "incrementing display" characteristic of 
group 4, and to some extent group 5, is a more sophisticated and efficient 
feature of sbnce calculators such as Rockwell. 

* 

u - 
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Table 10 gives a list of alternative actions associated with each of the 
first -6 productions. As can be seen, the five common categories resulted in 
three alternatives £or production P2 (P2A, P2B, P2C) , three for P'A (P4A, P4B, 
P4C), two for P5 (PSA, P5B) , one for P6 (P6A) , two for P7 (P7A, P7B) , and three 
for P8 (PSA, P8B, P8C) . In the process of developing models for the i4 mis- 
cellaneous subjects, several new alternatives were constructed as shown in " 
Table 10. Although a detailed analysis of the 'performance of each 
miscellaneous subject would require undue space, examples are given in this 
section. , For example, one of the novices gives the answers 2, 2, 5, 5, 12, 2, 
0, 5, 5, 12, 0, 0, 0, 3, 0, 0, 0, 3 for the 18 problems listed in Table 4, 
respectively. This subject seems "to evaluate expressions immediately when a 
number key is pressed, corresponding to subjects in group 3, However, in^ 
addition, this subject treats any irregular sequence of button presses (such as 
+ after +, or + after =) $s a reset or clearing of the display and register. 
The productions which describe this subject's performance are: P2B, P4B, P5C, 
P6D, P7B and P8B. As another example, dne of the experts gives the answers, 2, 
2, 3, 5, 7, 2, 4, 5, 10, 12, 2, 4, 8, 3, 4, 8, 16, 7, for the 18 simple 
problems, respectively. This performance is similar to model 1 except that the 
display is incremented for = afte.r +. The productions are P2A, P4A, PSA, P6A, 
PTAy P8C. Models were fit to each of the miscellaneous subjects by taking the 
best fitting common model (i.e., models 1 thtou^h 5) and changing as few 
^productions as necessary in order to make the fit perfetct. 
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The left side of the Table 10 lists the frequencies of each production 'for 
the experts and novices. There is a tendency for experts to rely on productions 
which involye incrementiog the display, i.e., P5B and P8C. For example, these 
productions are used 6 ^imes by novices and 14 times by experts. In addition, 
there is a tendency for experts to rely on productions which evaluate and dis- 
play for + and = but not for #, i.e., P2A, P4A, P7A, while novices tend to favor 
immediate evaluation and display for //, i.e., P2B, P4B, P7B. For example, the 

former set of productions is used 76 times by experts and 58 times by novices' 

•J 

but the latter set Is used 13 times by experts and 31 times by novices. ^ 
Analysis of Individual Subjects on Multiplication Problems 

The previous section encouraged the idea that it' is possible%to describe 
the subject's "model qf ' the calculator" for generating answers to 18 simple 
problems. These analyses were base^ on problems containing only six ;possible 
conditions. In the present section, we expand our analysis to include 16 
additional problems which contain multiplication. These are problems 11 through 
26 on Table 1. They provide three new conditions: x after # (i.e., pressing 
the multiply key after pressing a number key, such as 2 x) , // after x (i.e., 
pressing a number key after pressing ^a multiply key, such as 2 x 3) and = after 
X (i.e., pressing an equal key after pressing a multiply key, such* as 2 x . 
Also, up to this point we have considered only conditions which include two ' 
events, but this group of questions also allows us to explore whether subjects ^ 
use mote than two events to determine chains of arithmetic; for example, if a 
subject evaluated all multiplications before additions, then 2+3x7= would 
yi-eld an answer of 23 but if a subject evaluated chains in order of presentation 
then the answer is 35. Thus, this analysis will allow us to add three new 
productions . to each subject's model developed in the previous section, and to 
modify some productions for evaluating chain arithmetic. 
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For each subject, we assume that the six productions established In the 
previous analysis still are operating for problems 11 through 26. Thus, the 
goal of the present section is-/ simply to add three new productions (PIO, Pll, 
P12) to the model of each subject. All 16 problems contain the condition x 
after #; almost all of the problems cdntalh the condition, = after x; problems 
15, 17, and 23 provide the condition = after x; and several problems Involve 
chain of + and x operations such as problems 24 and 26. 

Table 10 shows the most common possible actions associated with each of the 
^ three new conditions (PIO, Pll, P12) explored in this section. As with the 
analysis of simple problems, one Issue concerns when an expression Is evaluated. 
If an expression is evaluated as soon as a multiplication operation (x) or an 
equals (==) key is pressed, analogous to Model 1 in the previous analysis, then 
the productions selected- would bei PlOB (no evaluate for #) , PllA (evaluate for 
x) , and P12A (evaluate for equals). If an expression is evaluated only when an 
equals key is pressed, analogous to' the delayed evaluation in Model 2, then the 
productions selected would b^: PlOB (no evaluate for #), PllB (no evaluate for 

) : 

x) , P12A (evaluate for «) . If an expression is evaluated as sooA as a number 
key is pressed, analogous to the immediate evaluation of Model 3, then the 
productions selected would be: PlOA (evaluate for #) , PllB (no evaluate for x), 
P12B (no evaluate for =) . Finally, if subjects used an incrementing display for 
evaluating expressions and numbers as in Model 4 or 5 in the previous section, 
then the selected productions would be: PlOB (no evaluate for #), PllB (no 
evaluate for x) , and P12C (increment display for . For purposes of this 
analysis we will refer to each of these four clusters of three productions as 
Model Im, Model 2m, Model 3m, and Model 4m, respectively. 
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Table 11 shows the answers generated by each of the f ou3>-multlplicatlon 
models (i.e., productions PIO,. Pll, and P12) . Thirty of the 33 novices and 23 
of the 33 experts generated performance on the multiplication problems that was 
consistent with one of the four models; however, specific answers to some . 
problems^ could differ from those listed in Table 11 in cdses where different 
systems for evaluating chains of arithmetic or different productions for P2 
through P8 were in use. The bottom of Table 11 shows the number of novices and 
experts who were fit by each of the four models. 
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Model 2m allows for )evaluation of a chain of arithmetic. Of the novices, 5 
subjects performed the arithmetic in order from left to right (as indicated in 
Table. 11), one subject . carried out additions before multiplications (e.g., 2 x 
3 + 7 = resulted in an answer of 20), one subject carried out multiplications 
before additions (e.g., 2+3x7= resulted in an answer of 23) and one carried 
out computations only on the last two entr'les in the register (e.g., 2 x 3 x 7 ■ 
resulted in 21; or 2 x 3 + 7 = resulted in 10). Of the experts, three^ out of 
five subjects showing model 2m performed chains from left to right, and two of 
the five experts performed multiplication before addition in a chain. 

There were also three miscellaneous novices and 10 miscellaneous experts. 
Of the novices, two subjects gave model Im answers for problems that involved 
only numbers, multiplication, and/or equals but model 2m answers for problems 
with numbers, addition, multiplication, and equals. One novice gave model 4m 
answers for problems with numbers, multiplication and/or equals but model 3m 
answers when problems involved multiplication, addition, numbers and equals. 
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Thus, these subjects behaved as If the conditions for actions depended on more 
than just the last two button presses. No additional productions vere con- 

structed-Nto try to fit this performance In Table 11, However, for the 10 unique 

( • \ • 

experts, several additional productions for PIO, Pll and P12 were constructed 

and are listed in Table 10, For example, one subject reset the display to aero 

for X after # and to no change for = after x. The productions for that subjett 

are PlOB, PllC, P12A, Another subject Ignored the equal sign when it followed 

the multiplication sign, a^ indicated by productions PlOB, Pll^, P12B, Two 

other subjects reset the display for « after x giving the productions P10B,j,* 

PllB, P12C. Thus, many of the experts tend to add new productions f or ,unusual*- 

button sequences; the effect of most of the new product^bns is sQme sort of ^ , 

"resetting" the display. The frequency of; use of each alternative production for 

1^10, Pll, and P12 for all subjects is suigmarized in the left side of Table 10. / 

Analysis of Individual Subjects on Complex Problems 

^ Finally, the performance o^ each subject on problems 35 through 44 was » ' 
analyzed.' These problems contain many of the condition,s already .described in 
the previous two sections; thus, it was assumed that each subject would use the 
twelve productions already determined by analyzing the first 34 problems* in the 
test. However ,9 probl^mfe^ 35 'tlvrou|h 44 aldo contain f(;iur new conditions: x after 
= , X after x, + after x, and x after +• Thus, in this section four new pro- 
ductions (P13, P14, P15 and P16) are added to the model of each subject, * 

Table 10 lists the» alternative productions for each of the four new con- 
ditions explored in this section. Table 12 gives some typical answers by 
subjects for problems 35 through '44. . ^ 
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Group 1 in Table 12 behave as if they were using productions 13A, 14A, 15A, 
and 16A along with delayed evaluation of expressions (based on earlier pro- 
ductions) • These subjects treat x after = and x after x as if there Is no 
change, and if two consecutive operation keys are, pressed (such as x after +) 
th^y use only the last operation that was pressed* As shown in the bottom of 
Table 12, there were ten novices and 15 experts who followed this procedure* 
There were also six more novices and one more expert in group 1'; these subjects 
behave as if they use the identical four new productions but they show Imlhediate 
evaluation of ' expressions when the number key is pressed* In addition, four 
novices, (and no experts) in group 2 behave like those in group 1 except that 
when there are two consecutive operations, the multiply "wins", t*e*, for 2x+ 
the register , stores 2x* The productions for this group are P13A, PlAA, P15B, 
P16A* Similarly, two more novices and one expert in group 2' use the same four 
new productions as group 2 but act as if an expression is evaluated as soon as 
a number key. is pressed* In addition, there were 2 novices and A experts in 
group 3* These subjects treat the four .new productions as if they serve to 
increment the display* This procedure 1$ Indicated by the combination of P13A, 
P14B, P15C, P16C* These -are the same subjects who increment the display for 
similar conditions such as + after + or + after « or after +• 

Thesre we^e also a large number of unique subjects — nine novices and 12 " ' 
experts* However, almost all of the subjects ari closely related to either model 
1 or model 3, with just one production slightly different* For example, one 
subject is like model 1 except that the display is reset to zero for x after + 
or « after x* The productions for* that subject aore P13A, PlAifL, P15D, P16D. 4 
Another subject has the same procedure as subjects in group 2 except that' x 
after x results in the display being multiplied by the register; the productions 
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tor this subject are P13A, P14B,>(P15B, *P4.6A.( For a third subject, the display 
Is Incremented for = after x and x ,af ter -* an^ when two consecutive operations 
are Input the multiply wins; the productions are, P13C, P14B, P15B, P16A. The 

4 ■> f 

left portion of Table 10 sunmiarlzes the frequencies of each of the alternative 
productions for all subjects on P13, P14, P15 and P16, 

General Conclusions 
Ch aracteri zin g th e Differences Among Subjects 

The present study suggests that subjects differ wlw^ respect to their, 
conceptions of the operation of electronic calculators. The foregoing analyses 
(summavized in Table 10) provided for a detailed description of the differences 
among subjects, with each subject being described as a list of productions. 
How*ever, the goal of this section is to provide a more integrated description of 
the major differences qmong subjects. Three basic kinds of differences were 
observed: (1) How is an expression represented in the register? For example, a 
series of key strokes such as 2+x3 can be represented as 2 + 3 or 2 x 3 or 0 or 
something else. (2) When is an expression evaluated? For example, does the 
, calculator evaluate at the earliest possible opportunity such as 2 + 3 resulting 
in a display of 5, does the calculator wait for an equals to be pressed before 
an evaluation takes place, or does the calculator compromise between these two 
extremes? (3) How is an expression evaluated? For example, a chain of arifh-- 
metic lll;e 2 + 3x7 can be evaluated from left to right (answer is 35) or with 
multiplication tirst (answer is 23) or in some other way; furthermore, , non- 
standard sequences such as 2+« can be evaluated by Incrementing the display to 4 
or by ignoring the plus (display is 2) or by reseting the display to 0. In «thls 
section, we explore how the subjects differ with respect to their conceptions of 
how to represent expressions, when to etfaluate, and how to evaluate. 
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Standard conditions . First, there are some general differences which " 
eine^ge-4>y^^investigating differences for standard conditions such as # after +^ 
// after x, + after //, x after = after //. These are sequences that follow ^ 
the standard grammar of arithmetic, and are listed as P2, PIO, P4, Pll, and T7 
in Table 10. - . . 

The first issue of how to represent expressions is fairly straightforward 
for all subjects — symbols are added to the register in order 'from left to right. 
Fbr example, the keystrokes 2 + 3 x 7 is represented in exactly that way in the 
register. ' ' 

The second' issue is when to evaluate the expression. In our analysis we 
located three distinct approaches to the question of when to evaluate. The 
compromise metho^ is to evaluate an expression whenever an equals key or an 
arithmetic operation key is pressed but not when a number key is pressed; the 
immediate method is to evaluate as soon as a number key is pressed (e.g., for 3 
+ 5 display shows 8); the^layed method is to evaluate oriiy when an equals key. 
±s pre^ssed (e.g., for 3 + 5 th^^isplay shows 5). The novices and experts tend 
to differ with respect to their consensus on when to evaluate. Of the novices 
13 tend to opt for compromise evaluation, 13 for delayed evaluation, and 7 for 
immediate evaluation; for experts -there is a much stronger consensus of 2A 
subjects favoring compromise evaluation with 7 favoring delayed evaluation and 2 
favoring immediate. A ch^-square test ^revealed that novices and experfs differ 
significantly with respect to tile proportion of subjects favoring compromise 
evaluation, = 6.15; df ='l, £ < .05. ^ 

The third fssue concerns how to evaluate an expression. In most cases, 
subjects overwhelmingly follow the norma|L rules of arithmetic. However, as * - 
noted earlier, when subjects use delayed evaluation they may.be confronted with 
a chain of arithmetic soch as 2 + 3 x 7". While tlte majority of subjects eval- 
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uate a chain from left to right (i.e., generciting an answer of ^35) , some experts 
carry out multiplication beforfe addlflonv^l, e • , answer is 23), and some novices 
use other schemes such as* carry out additions before multiplications or carry 
out only the last computation (i.e., answer is 21). * . i 

Non-standard conditions (equals after operation) . In the present study we 
also inves'tigated subjects' Interpretations of several non-standard conditions^ 
such as = after + after Xi The§e are conditions* which violate the grammar 

that' demands ffiiiumber between the operation symbol and the equals symbol. Table 
10 represented, t^iese as P8 and Pf2. 

The main issue here is how to* represent and evaluate an expres'sion when the 
last key press was an operation and now an equals is pressed. For production 
P8, the majority of novices (n = 27) and ttie majority of experts (n « 25) ignore 
the last + key that was pressed. ThuJs, a sequence llk^ 2+= results in a display 
of 2, or a sequence like 2+3+= results Iri a display of 5. We*call this '.the "no 
effect" app'roach because subjects act^^as if the plus key had no effect. A 
second approach is what we call the "incrementing" approach; here subjects 
create some number to go between the + and the « such as the number in the^ 
display. For example, t» the sequence 2+= results in a display of 4 (I.e., it is 
treated as 2+2«) , or 2+3+= may result in 10 (i.e., it is treated^as 5+5«) or 8 
(i.e., it is treated as 2+3+3=^). There were* three |nQ.vices and 6 Experts wlio 
opted for the InQl^menting approach. A third option is what we call the "reset" 
approach. Here subjects reset the display to some number (such zero) for any 
non-standard sequence .oF key presses. Three/novices and two experts used a 
version of the reset approach. The iiompax'able figures for production P12 were 
29 novices and 2'^ experts favored the no effect approach while 4 novices and 8 . 
experts favored tl^ Incrementing approach. Although the proportion ^bf subjects 
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favoring the incrementing approacH^ i^* twice as^'-high for experts as for fioVlces, 
chira4^are tests failed to iricflcate that' the pjropoi;tlon of Incrementing subjects 
vas greater for experts '^in P8, = df = 1, br P12, = .92, df I. 

Non-standard conditions (two consecutive .operations) > Another type of non- 
Standard condition^ investigated in this study was two consecutive operations 
such as + after ^, x after x, + after x, or x, after +. These are conditions 
which violate the grammatical demand for a number between any two operation 
.symbols • Table 10 presents these as P5, P14, P15, P16.' V ^ 

The main issue here is how to represent and/or evaluate an expression when 
the last key presses are two arithmetic operators, the three major options 
c]}osen by our subjects correspond to those discussed above: "no eff^t" in- 
volves selecting one of the two operation signs to be included in the register 
and ignoring the other jv for example, the most common version of this approach is 
to ignore the^ seeond^ operation so that 2++ is represented' in the register as 
2+ or 2x;x is represiented as 2x. "Incrementing" involves selecting a number to 
be inserted between the operator symbols; the most common version of .this — • 
approach is to insert the nlimber from the display so that',2++ becomes 2+2+ or 
2xx becomes 2x2x. "Reset" involves clearing the display such as 6'etting it to 
zero; for example, 2++ results ^in 0 being' displayed. '';'For production P54:he ' ^ 
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tjority of novices (n = 28) and^ experts (n = 24) opted for the no effects 



approach; in addition four novices and nine expdrts opted for the incrementing;^ 
approach; and one novice and no experts reset: the displ'^. The figures for 

^production P14 are similar: *28 novices and 2A expewrts^ opted for no effect; 
incrementing was opted for by 4 novices and 8 experts; and one novice and one - 
expert opted for the reset approach. The patterns for P'l? and P16 are similar—- 

^the majority of each nOvice^I^d experts opt for no effect but ^ substantial 
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xiumber of experts opt for the incrementing option. In all productions, the 
proportion of experts who increment is more than twice that qf the novices, . 
However, even in- the inpst extreme case, the" differences in proportfdn of "in- ^ 
trementers" between experts* arid novices fail» to rd'ach statistical significance, 
= 2.27; df = 1. . ' , ^ ^ ' : ^ ^ 

Nori-standaiyi 'Conditions (operation after equals) . Finally, o^r test in- 
volved two productions P6 and P13, which involve + after = and x after « re- 
spectively. The status of the rpgister after -an = key is pressed is that it 
contains a number i -Thus, these rion-standard conditions are most ,f requen^ly 
treated in the 9ame way that + after # or x , after # is treated. For P6, 30 of 
the novices and 32 of the experts use this "no effect" ^approach of simpAy adding 
a plus sigh tb the regi3t:er. For P13, 28 novices and. 27 experts, follow the "no 
effect" approach of adding a multiply Bign to'the register. However, a sizable 
minority of the experts (n = 6) opt for an incrementing approach while a sizable 
minority ot the novices opt for a reset option (n = 4). ^ 

Summary . Thfe present .«tudy provides new inf orntation concerning how humans 
think, about calculators. First, we were able to apply the analytic techniques 
of cognitive *i)sychology to a real-world domain. This allowed a formal anci" 
detailed description of h&w each subject Interpreted what was going on Inside 
the "black box" when a key was pressed. Second,* in spite of the fact that all 
of our subjects were proficient in using a calculator to 6olv^ st^andard compu- 
tational problems,, we observed tremendous individual differences among user3 in 
their interpretations of the logic of the calculator's operating system". Thus, 
in 3pite of apparent similat performance on standard problems, people differ 
greatly in their knowledge of fiiow tfee calculator solves problems. 
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Experts tended to give more tfbnsistent answers, as would be expected; 

\ . . - , ■ 

however, they also tended to prefer certain operating characteribtics such as 

evaluating an' expression for either an operation key or an 'equal key (cotopromise 

e/aluator) , and incrementing the display during evaluation with non-standard 

conditions. Further work is needed to determine whether people with certain 

sets of intuitions can use their calculators more creatively or can learn a new 

computer language ^such as programmable ca,lculators or BASIC) more efficiently 

.than people with other sets of intubations . IriTaddition, future work is needed . 

to determine whether intuitions—once th6y havexbeen diagnosed — can be altered 

^through instruction. It is hoped that the groundwork laid in this study will 

serve as an incentive for continued work in the development of a theory of 

computer literacy. 
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Reference Note 



1. Suydam, M. N. State of the art reviewon calculators: Their use in 

education. Columbus, Ohio: Calculatpr' Information Center, Report No. 3, 
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^Twelve gublects were from the University of Pittsburgh and 34 subjects were 
from the University of California, Santa Barbara. However, since there were 
no systematic; differences between these groups in age, sex, CPA, SAT scores, 
or answers to the experimental test, they have been combined into one larger 

sample. , * 

\e wish to thank Dr. Larry Lichten of the Computer Science Department^of 

' the University of California, Santa Barbara,! for his help in locating subjects 
for study 2. 

^For each problem in which the subject gave inconsistent answers between 
set A and set B, the answer to the A set was used unless it was Inconsistent 

with related Items. ^ 

*Slnce the HP-21 uses reverse Polish notation (RPN) it was not used as a 

model In this study. 
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Table 1 

Frequei^cies of Answers for 44 Problems 






Problem 

Number- < 


Problem 


Stijdy 1 
Modal Answer^ Alternatives 


Study 
Modal Answer 


2 • 

Alternatives , 


1 


2 






' (2)-33 




2 


2+ 




- 


(2)-33 

^ f 


(0)-i ■ 


3 


2+3 




(5)«8 


(3)-32 


(5)-l 


4 
5 


2+3+ 

r 

2+3+7 • . 


(5)-2r 

• C7)-26 


/ ( 

— ' (3)-12 

(12)=6;(15)-1 


(5)-23 
(7)-32 


(3)-9;(0>-i 
(12) -i 


6 




(2)-33 




(2) -33 




7 


2+= 


(2)=29 


(4)-3;(0)-l 


(2)-27 


(4)-6 


8 


2+3= 


(5)=33 




(5)-33 




9 


2+3+= 


(5)-30 


(lO)-3 


(5)-26 


(l0)-5;^)-l;(8)-l 


10 


2+3+7" 


(12)-29 


(10)-2;(11)-1;(15)-1 ' 


1 





(12) =33 



11 
12 
13 
14 
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2x 
2x3 
2x3x 
2x3x7 
• 2x» 



(2) =33 

(3) «24 

(6) -24 

(7) -24 
(2) -29 



(6)=9 

(3) -9 
(42) -9 

(4) -4 



(2) -32 

(3) -33 

(6) -23 

(7) -32 
(12)-25 

(6)*33 



(0)-l 

/ 

(3)-9;(0)-l 
(6)-l 



(to 
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Problem 
Number 






Frequencies of Answers for 44 Problems 




- 

Problem 


- 

Modal 


,> Study 1 » 
Answer Alternatives 


- 

Modal Answer 


Study 2 

Alternatives 


17 


2x3x= 


(5)-29 


(36)-4 


" (6)-23 


(36)«6;,(3)«2;(18)-1;(0)»1 


18 


2x3x7= 


(42)-3]. 


(21>-1;(7)=1 ^ 


(42)-32 


(41)-1 


19 


2+3x 


^(5)-22 


(3)=11 


(5)-18 


(3)-13;(0)-l;(6)-l . 


20 


2+3x7 


(5)=25 


(35)=7;(42)=1 


(7)-32 


(42)'-l 


21 


2x3+ 


(6)-24 


(3)-9 


(^)-25 


(3)-7;(0)-l 


22 


2x3+7 


(7)=26 


(13) -7 


(7)-32 


(13)-1 - . 


23 


2+3x- 


(5)-26 


(25)-3;(3)-2;(ll)-l;(2)=l ' . 


(5)-23 


(25)-3?(3)-3;(ll)-2;(2)-l;(15) 


24 


2+3x7- 


(35)-29 


=i(23)-2;(21)-l;(42)-l 


(35) -2 3 


(23)-7;(42)-2;(45)-l 


25 


' 2x3+- 


(6)-29 


V (12)-3;(3)=1 


*(6)-26 


.(12)-5;(3)-l;(18)-l ' 


26 


2x3+7- 


(13)=29 


(42)=2;(10)»l;(20)-l'^ 


(i3)-3r 


(ll)-l;C42)-l 



27 
28 

30 
31 
32 

er|c* 



2++ 


(2)=29 


2+=-fc- 


(2)=27 


2+-+=+ 


(2)=26 


2 +-+3 


(3)-28 


2++« 


(2)-27 


2+-+- 


(2)-28 


(8)-5;(4) 


-3;(6)-l 
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(4)^3J'>(0>-1 

(4) -4;j(e)-2 

(8)/3;(0)=2;(4)-l;(6)-l 

(5) [ 

(40-3>f^)-2;(0)-l 
(8)-2;(4)-L;(6)-l;(0)-l 

c 



(2) -2 3 
(2)-25 

(2) -23 

(3) -32 
(2)-22 



(4) -9; (0)^1 " 
(4.)-8;(0)-l 

(8)-5;(4)-2;(6)-2;(0)-l 
(2)-l 

(4)-6;(8)-4;(6)-l 
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. Table 1 (Con\t.) 








• 




• ^ Frequencies of Answers for 44 


Problems 




Problem 


• 




Study 1 " 


• 


Study 2 ' 


Number 


Problem 


Modal Answer Alternatives 


Modal Answer 


Alternatives 


33 




(2)=28 


(16)«3;(8)»1;(6)=1 


" (2)-24 


(16)-5;(6)=3j(8)-l 


34 




. .(5)»26 


• 

(7)=4;(3)-3 


(4)«26. 


(7)-6;(3)-l 


35 


2xx 


(2)«29 




(2)-23 


(4)-9;(0)-l 


36 




(2)-28 




(2)-24" 


(4)-8;(0)-l, 


37 


2x*x«x 


(2)"27 




(2)=21 


.(16)-5;(8)-4';(4)-2;(0)-l 


38 


2x«x3 


(3)-24 


(6X-9 


(3)-29 


(6)-3;(12)-l . 


• 

39 


2xx= 


(2)=26 


(4)«4;(8)-lj(P)«l;(l^)-l 


(2)-20 


- ■ (4)-7;(l6)=3;(8)=2;(0)»l 


40 


2x=x» 


(2)-27 


(i6)-3;(0)-l;(4)«l;(8)-l 


(2)-22 


(l6)-5;(8)-3;(4)-2;('0)-l 


41 


2X«X«X*Sr 


(2)=27 


(256)=3(8)-l;(16)-l;('0)-l ' 


(2) -21 


(256)-5; (l6)-2; (8)-3;(32)-l; (0)-. 


42 . 


2x»x3» 


(6)=24 


(3)-5;(12)-3;(48)-l 


(6)-24 


(12)-7;(3)-l;(0)-l 


43. 


2x+3» 


(5)-l6 


(6)=13;(7)»2;(8)-1;(3)-1 


(5)-21^ 


(7)-6;(6)-3;(e)-2;(3;-l 

1 


44 


' 2+x3«. 


(6)«25 


(5)-4;(3)-2;(12)-2 


(6)-24 


(12)=6^(5)»l;(7)=l;(e)-l 
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Number in parentheses indicates answer; number to right of equals indicates frequency. 
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Table 2 



Name 



Condition 



Conditiony^f or Simple Problems 
. Example Description 



PI- 




after 




2 


3- 


Pressing, a ^nijmber key after pirie^ssing a number, key • 


( P2* 




after 


+ 


+ 


3 • 


^Pressing a number key after pressing a pj.us key. 


P3 




after 




SB 


3 


Pressing a lu er key after pressing an equals key. 


P4* 


+ 


after 




2 ' 


+ 


Press ing^a^plus key after pressing a number key. 


P5* 


+ 


after 


+ 


+ . 


+ 


' Pressing a plus key after pres3ing a plus key. 


P6* ' 


+ 


after 






+ 


Pressing a plus key after pressing an "fequa Is key . . 


P7* 




after 




3 




Pressing an equals key after pressing a ntimber key. 


P8* . 




after 


+ 


+ 


ts. 


Pressing an equals key aJ^ter pressing a plus key. 


P9 




after 




as 




Pressing an equals key after pressing* an equals key. 



Note. — , 



■Asterisk (*) indicates that production is relevant to simple problems. Pi, P3 and P9 ate not 
relevant since they do not occur in the simple problems. 
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, ' ■ ' • - -X' ' ■ ' . 

Some AclSLon for Simple Problems > , 

■ . , •. - y V ■ 

Action ^ Description . I ' a * s I ' 

D « D No change lij^'the display . .\* . ' ^ • 

IT « # Erase the old number, from the^ display. Put the new number In th6 display. 

D » R Erase the old number from the display. Copy the number from the riegister Into the display. / ^ ^ 

D « eval(R) -Erase the old number from Jhe display. Pat the value for the'^xpresslon In the register Into the display. 

. ^ * ^ ^" • • • ^* 

D « eVar(D+R) Erase the old number from the display. Replace It with the value fo?r the sum of that number from the 

display and the value In tfle register. 

, . *^ ' • ■ « 

R »Jl''* No change in register;* ' ^ ' ^; . 

R « # Retain the present expression in the register. Place a number to »the right 'of the expression in the register^ 

R » "R+//" Retain' the present expression in the register. Place a number to the right of th€i expression in the register. 

r <7 . , ' ' 

R'« eval (R) Erase the old number or expression from the register. Replace it -ith the evaluation bf the number dt ^ 

^ . ^ expression. 

R « eval (D+). Erase the old number or expression from the register. Riiplace' it with the sum of the number JLn the 

display plus the e^filiiatlon of 'the register. 

R - eval (R)+ Erase the old expression or number from the register. Replace it wi,th the evaluation of that number or 

expression, and follow that with a plus. 

• ^ ^ . . ■ ■ * ■ S? 
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' * - Table' 4 / 

Problems and Answers ^^r 18 Simple Items for Four Groups of Subjects 



rroDxeci 




\:rrOUp X 






\J V/ %A • 


Group 5 


Niunber 


Problem 


Answer 


Answer 


Answer 


Answer 


- Answer 


1 


2 


2 • 


2 


2^ 


/ 

2 / 


r 

2 


2 


J2+ 


2 




2 


2 , 


2 


3' 


2+3 ' 


3 


. 3 ■ 


5 


3 


3 


• 4 


" " 2+3+ 


5 ■ 


3 


5 


5 • 


> 5 


5 


2+3+7- 


'7 


' 7 ^ 


12 


7 


7 


6 


2= 


2 


: 2 ■ 


2 


2 


2 


7 


2+= 


' 2 


2 






2 


8 


2+3= 


5 


5 


5 


5 


5 ' 


9 


2+3+= 


B 


5 


5 


10 






2+3+7= 


12 




12 


12 


12 


27 


2++ 


2 


2 


2 


4 


4 ■ 


28 


2+=+ 


2 


2 


2 


4 


2 


'29 


2+=+=+ 


2 


2 


2 


8 




30 


2+=+3 


3 


3' 


5 


3 


3 


31 


2++= 


2 


2 


2 


8 . 


• 4 


32 




2 


2 


2 


8 


2- 


33 




2 


2. 


2 


16 


2 

t 


34 


2+=+3= 


5 

1 


. 5 


■ -v 


' ■ 7 


' 5 


Number of Subjects- 














Study i 


8 


• 10 


, 5 


2 • 


1 


Number of Subjects- 














Study 2 


11 


7 


2 


4 


3 



Miscellaneous 



V 



ERIC 



V 



Table 5 

Production System for Model 1 (Compromise Evaluation) 



Production 
Name 

P2A 

P4A 

PSA 

P6A 

P7A 

PSA 



Conditj.on 
If # after + 
If + after # 
If + after 
If + after « 
If « after // 
If « after + 



Action 




















then 


Set 


D 


«8 


# 




Set 


R 


ei 


"R + 


#" 


then 


Sett 


D 


an 


eval 


(R) 


Set 


R 




eval 


(R) + 


then 


Set 


D 


XT 


D 




Set 


R 


+ 


» R + 


then 


Set 


D 


3S 


eval 


(R) 


Set 


R 


m 


eval 


(R) 


then 


Set 


D 


ae 


eval 


(R) 


Set 


R 


wt 


eval 


(R) 


then 


Set 


D 




eval 


(R) 


Set 


R 


m 


eval 


(R) 



Comments 



(NO CHANGE) 



Note. — Evaluates after + or ». Display is non- incrementing. 



1:^5 



ERIC 



4 



Table 6 

Production System for Model 2 (Delayed Evaluation) 



Production 
Name ^ 

P2A 

P4B 

PSA 

P6A 

P7A 

PSA 



Condition 
If # after + 
If + after # 
If + after + 
If after « 
If ^ after # 
If » after + 



Action 

then Set' D « # 

then Set D « D 

,then Set D » D 

then Set D » D 

then Set D « eval (R) 

then Set D « eval (R) 



Set R - "R + #" 
Set R » eval (R) + 
Set. R + ■ R + 
Set R ■ R + 
Set R " eval (R) 
Set R - eval (R) 



Comments 

(SAME AS MODEL 1) 

(SAME AS MODEL 1) 

(SAME AS MODEL 1) 

(SAME AS MODEL 1) 

(SAME AS MODEL 1) 



Note. — Evaluates after =. Display is non- incrementing. 
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Table 7" 

Production System for Model 3 (Immediate Evaluation) 



Production 
Name 

P2B 

P4B 

' PSA 

P6A 

P7B 

P8B 



Condition 
If # after + 
If + after # 
If + after + 
If + after - 
If » after # 
If » after + 



Action 



Comments 



then Set D - ev&l (R +#) Set R - eval (R+#) ^ 

then Set D - D Set R - eval (R) + 

then Set D - D Set R - R + , (SaMe AS MODEL 1) 

then Set D - D Set R - R + (S^ AS MODEL 1) 

Set R - R ' (NO CHAlsGE) 

Set R - R (NO CHANGE) 



Set D-D 



thein Set D «■ D 



Note, — Evaluate after #. /Display is non-incrementing. 



123 



vn 



Table 8, 

Production System for Model 4 (Incrementing Display) 



Production 



Nximber ^ 


Condition 




Action 














Comments ^. 


P2C 


If # after 


+ 


then 


Set JB * 


# 




Set 


R 


■i 


eval (R + #) 




P4C 


If + after 


# 


then 


Set D - 


R 




Set 


R 


■i 


R 




P5b' 


If + after 


+ 


then 


Set D « 


eval 


(D+R) 


Set 


R 




eval (D+R) 


(INCREMENTING DISPLAY) 


P6A 


If + after 




then 


Set D - 


eval 


(R) 


Set 


R 




eval (R) 


(SAME AS MbDEL 1) 


P7A 


\ If - after 




then 


Set D - 


eval 


(R) 


Set 


R 




eval (R) 


(SAME AS MODEL 1) . 


P8C 


If ■ after 


+ 


then 


Set EL^ 


eval 


(D+R) 


SEt 


R 




eval (D+R) 


(INCRkMENTING DISPLAY) 



Note. — Evaluate after + or «. Display Is Incrementing, 



— c 

H 

CO 
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Table 9 

Pro4uctfon System for Model 5 (Partially Incrementing DisplAy) 



Name 


Condition 


Action 




4 






Comments 


P2A 


If # aft6r + 


then Set D * 


t 


Set R - 


"R + 


#" 


(SAME AS MODEL 1) 


/ P4A 


If + after # 


then ^Set D - 


eval (R) 


Set R - 


eval 


(R) + 


(SAME AS MODEL 1) 


1^ P5B . . 


If + after + 


then Set D - 
• 


eval (D+R) 


Set R - 


eval 


(D+R) 


(SAME AS MODEL 4) 


If + after - 


then Set D - 


eval (R) 


Set R - 


eval 


(R)' 


(SAME AS MODEL 1) 




If » after # ^ ^ 


theri Set D - 


eval (R) 


Set R - 


eval 


(R) 


(sMe as MODEL 1) 




If » after + 


then Set D « 


eval (R) 


Set R - 


eval 


(R). 


isM. AS MODEL 1) 



Note.-*rCompromise between model 1 and model 4. 



Table 10 

^ Frequencies of Productions for All Subjects 



Frequency frequency Prdduction 

" '■» 

In Study 1 in Study 2 Number Condition 



21 



27' 



P2A If # after + 



P2B If* after + 



P2C If # after + 



P2D- If # after + 



Actibg ^- ' 

them Set D - #, 

jset R "R+Jl^' 
ther Set P « eval (^#) , 

Set R « eval (R+#) 
then Set D « #, 

Set R « eval (R+#) 
then Set/D « eval (R+D) , 
Set k - eval (R+D) 



Description 

Delayed evaluation and display 

Immediate evaluation and display 

Innnediate evaluation and delajred display 

Innnediate evaluation and. display with 
incrementing evaluation^ 



11 

18 

4 
0 

28 

ERIC 



18 



A 

2 

24 



134 



P4A If + after # 

P4B If + after # 

P4C If + after # 

P4D If + after # 



(p^ 



PSA 



then Set D » eval (R) , 

Set R » eval (R)' + 
then Set/D » D, 

Set\R - eval (R) + 
then Set D ^ R, Set R-R+ 
then Set D - 0, Set R - 0 



Immediate evaluation and display 



Immediate evaluation , and delayed display 



Delayed evaluation and display 



Reset to zero 



If + after + 



then Set D-D, 
Set R+ • Jl+ 



No change 



o 

s 8 



J.35 



Table 10 (continued) 



Frequency Frequency Production 

in Study X in Study 2 Number Condition Action 

4, • 9 \, P5B If + after + then Set D « eval (D+R) , 

Set R « eval (D+R) 
1 .0 P5C If + after + then Set D « 0, 

' Set R » 0 . 



Description ' - . 

C^Bfinediatfe incrementit(| evaluati<Jli and 
display 
Reset to 0 



30 



0 ' 



32 



P6A If + after » then Set D-D, 

Set R « R+ 

p6B If + ^fter = then Set D « D, 

^ * Set R » eval (D+R) 

P6C If + after « then Set D « 0, 

Set R R+ 

P6D [ If + af4:er » ' then Set D « 0, 

Set R « 0 

.■i 

P6E * If + after - then Set D « D, 

. . ^ - Set R « eval (#+D) 



No change ^in display, add + to 

expression in register 
No change in display, immediate 

incrementing evaluation 
Reset display to 0, add + to 

expression in register 
Reset display and register to 0,* 

evaljiate sum of constant aiid value 

in display 
No change in display \ 



< 



Frequency Frequency Productl' 
In Study 1 In Study 2 Number 



26 



21 



ERIC 



31 



23 



P7A 



P7B 



P7C 



l3o 



Condition 



If « after 



Table 10 (continued) 



Action 



then Set D - eval (R) , 
Set R « eval (R) 
If « after # then Set D » D, 

Set R « R 

If « after # then Set D » R, 

Set R « R 



PSA If « after + then Set D = eval (R) , 

Set R * eval (R) 

P8B If « after + then Set D « D, 

Set R » R 

P8C If = after + then Set D » eval (D+R) , 

Set R « eval (D+R) 

P8D If » after + then Set P « eval (R) , 

Set R « 0 

PBE If « after + then Set D « ^val (D+#) , 

Set R * R 



Description 

Immediate evaluation and display 

I ■ ^ ♦ 

No change 

Display value in register 



Immediate evaluation and display 

No dhange in display or register 

* . _ 

Immediate incrementing evaluation 

^ and display 

Diajplay the evaluation of the expression , 

I 

j in the register ^ reset the o 

I ■ ' 

J register to 0 
Display the sum 6t the value in the 

display plus a constaltt, no change 
in register 139 



8 



Table 10 (continued) 



11 



19 



15 



16 



19 



18 



frequency 


Frequency 


Production 








lii Study 1 


in Study 2 


Number 


Condition 


Action 


Description 


1 


2 


P8F 


ir » after + 


then ^et « D, 


No change in display » set register to 6 










Set « 0 


im ^ mm ^ mm ^ ^ mm mm- ^ w» mm ^ -m^ "^mi 


.8 


6 


PlOA 


If # after x 


then Set D « eval (R*#) , 


Immediate evaluation and display 






/ 




Set R - eval (R*#) 




25 


27 


PlOB 


If # after x 


then Set D « #, 


Delayed evaluation and display 




e 






Set R - R*# 





PllA If X after # then Set D - eval (R), 

Set R - eval (R)* 

pilB If X after # then Set D-D, 

Set R « eval (R)* 

Pile If X after # then Set D « 0, 

Set R » R* 
(other mixed) ^ 



Immediate evaluation and display 

f 

Immediate evaluation and' no change in 
display 

Delayed evaluation and reset display to 0 



P12A If « after x then Set D - eyal (R) , 

Set R - eval (R^ 



Immediate evaluation and display to o 



14i 



Table 10 (continued) 



Frequency Production 



study 1 


In Study 2 


Number 


Condition 




Action 








Descriptioxt 


7 


7 


P12B . 


If - after 


X 


then Set 


D 


- D, 




No change ^ ^ 












. Set 


R 


- R 


* 




4 




P12C 


, If " after 


X 


then Set 


D= 


" eval 


(D*R) , 


"V, , ' ■ " 1 

Immediate incrementing evaluation and 












Set 


R 


- eval 


(D*R) 


display 


0 


1 


P12D 


If « after 


X 


then Set 


D 


" eval 


(D*R), 


Immediate incx;ementlng display » Immediate ^ 












Set 


R 


« eval 


(R) 


evaluation for Register 


0 


2 


P12E 


If « after 


X 


then Set 


D 


" eval 


(R*D), 


Immediate evaluation and display in the 












Set 


R 


« eval 


(R)* D 


constant increment 



(other mixed) 



28 

4 
1 



0 

ERIC 



27 

0 
3 



2 
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P13A 

P13B 
P13C 

P13D 

P13E 



, If X after » then Set D ■ D, 

Set R » D* 

If X after - then Set 6 « '0, Set R - 0 

If X after - then Set D - eval (D*R) , 

Set R - eval (D*R) 

If X after - triien Set D - eval (D'^), 

^ ; , Set R - R 



If X after 



then Set D " R» 

Set eval (f)*R) 



Delayed evaluation and display 



Reset to 0 

Immediate^ incrementing evaluation and ^ 

display * * g 

Immedia^te incrementing display ^ no change g 

m 

in register ■. r: S 

Immediate incrementing evaluatlcmff 

. delayed display 143 



Table 10 (continued) 

Frequency Frequency Production 

In Study 1 in Study 2 Number Condition Action 



28 



24 



8 



P14A If X after x then Set D - 

Set ^ - R* 

P14B If X after x then Set D * eval (D*R) t 

Set R - eval (D*R)* 

P14C If X after X'. then Set D - 0, 

Set R « 0 

P14D If X after x then Set D « D, 

Set R - D*R 



Description 

Delayed evaluation and display 
« 

Immediate incrementing evaliiation and 

display 
Reset to 0 

Immediate incrementing evaluation, 
no change in display 



19 



10 



25 



I 1 



P15A If + after>X then Set D-D, 

Set R " R+ 

P15B If + after x then Set D « D, 

Set R « R* 

P15C If + after x then Set D - eval (D*R)t 

Set R - eval (D*R) 

P15D If + after x then Set D - 0, 

Set R - 0 > 

P15E If + after x ^ then Set D ■ eval <D*R) , 



Set register sign from * to + 
No change 

Immediate incrementing evaluation and 

display 
Reset to 0 



a 

^ c 



Immediate incrementing evaluation and 



14® 



Table 10 (Continued) 



frequency Frequency ' Production 

in Study 1 In Study 2 Number Condition - Action 



28 



2 
0 



25 



1 

3 



P16A If X after +. then Set D-D, 

. " Set R » R*. . 

P16B If X after + then Set D r D, 

Set R - R+ 

P16C If X after + then Set D ■ eval (If*R) , 

k Set R - eval (D*R) 

P16D If X after + then Set D « 0, ^et R - 0 

P16E If X after + then Set D » eval (D*R) , 

Set R - eval (D*R)* 



Description 

Set register sign from + to * 
No change , 

Immediate incrementing evaluation and 

display 
Reset to 0 

immediate incrementing evaluation and 

display with register ^ sign set to * 



146 



147 
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Table 11 
*■ » * t '* 

Problems and Answers for i? Multipliction Jtems by Four Groups of Subjects 



Emblem 


• 


Group Im 


Group ' 2m 


Group 3m 


Group 4m 




Number 


•Problem 


Answer 


Answer 


Answer 


Answer 


Miscellaneous 


11 


2 X 


. 2 ' ' 


■* 

' 2 


2 


2 




12 


2x3 


3 


3 


■ 6 


3 




13 


2 X 3 X 


6 


3 




• 6 


• 


14 


2x3x7 


7 


"7 


42 


7 




15 


2 X - 


2 


2 


2 




















16 


2x3- 


6 


.6 " . 


' 6 . 


6 




• 17 


3 X - 


' 6 


'6 . , 


6 


36 




, IS- 


2 X 3 X 7 - 


42 


4? 


42 


42 




IS ■ 


<2 + 3 X 
>• 


5 


3 


5 


5 




20 


2 + 3x7 


7 


7 


35 


7 




21 


2x3 + 


6 




6 


6 • 




22 


2x3 + 7 


7 


7 


13 


7 




23 


2 + 3 X " 


5 


5 ' 


5 


25, 




24 


2 + 3x7- 


35 


35 


35 


35 




'25 ' 


2 X 3 + » 


6 


6 


6 


12 




26 


2x3*7- 


13 


13 


13 


13 




Number of Subjects-Study 1 11 ' 


8 


7 


5 


3 


Number of Subjects-Study 2 15 


5 


0 


3 


10 
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. Table, 


12 






> 


Problems and Answers for iO Complex Items by Five Groups of Subjects , 


Problem 




Gfoup 1 


Group 1' 


Group 2 


Group / 


broup J 




Problem 


Answer 


■ Answer 


Answer^'' 


Answer 


AnswcBT Miscellaneous 


35 


2 X X 


2 


2 . 


2 


- 2 ' 


4 " , 


36 


2 X a X ^ 


2 


2 




2 




37 


2 X «=S5X;= X 


2 


'2 . 


2 


2 


16 


38 


X = X = 3 


3 


6 


• 

3 


- 6 . 


3 




3 X X » 


2 


2 


2 


2 




ft 


2 X = X « 


2 


2 


' 2 


2 


16 


41 


2 X «= X » X = 


2 


2 


2 


2 


256 ' 


42 


'2 X « X 3 « 


6 


6 


6 


6 


. - a ^ 

12 

t 




2 X + 3 = 


5 


. 5 


6^ 


6 






2 + X 3 « 


■ 6 


6 


6 


6. 


12 

^ 


Number of 


Subjects-Study 1 


10 


6 


4 


2 




Number of 


Siibjects-Study 2 


15 




0 


1 


4 12 












V 








> 


i 
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CHAPTER 4 . ' 

TASK ANALYSIS, DEVELOPMENT OF EVALUATION INSTRUMENTS, 
AND DIAGNOSIS OF BUGS FOR THE BASIC LANGUAGE 



/ 

This 



This 



Note ; 

' , .•■ • • 

article has been published as the following citation: 

Mayer, & Baytnan, P. Users' mental models of BASIC. Technical 

Report No. 82-2. Santa Barbara: University of California, 1982. 

article has been submitted for publication as follows: 

Mayer, R. E., & Bayman, P. Users* misconceptions t)f BASIC programming 

statements following hands-on experience. Cognition & Instruction , 

under review, 

research was also presented at a professional meeting as the following: 
Mayer, R. E. , & Bayman, P. Beginning programmers' comprehension o^ 

BASIC statements. Psychonomic Society, Minneapolis, November 11-13, 
1982/ 



This 



I5u 
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t Abstract 

In the process of learning a computer language, beginning programmers may 
develop mental models for the language. A mental model refers to the user's 
conception of the "Invisible" Information processing that occurs Inside the 
computer between Input and output. In this study, thirty undergraduates ^ 
learned BASIC through a sfilf -paced, mastery manual and simultaneously had 
hands-on access to an Apple II computer. After Instruction, the students . 
were tested on their mental models for the execution of each of nine BASIC 
statements. The results show that begl^ining programmers — although able' to 
perform adequately on mastery tests in program generation — possessed a wide 
range of misconceptions concerning the statements they had learned. For 
example, the majority of the beginning programniers had either Incorrect 
conceptions for or no conceptions of statements such as INPUT A, READ A, and 
PRINT C. This paper presents a catalogue of beginning programmers' 

conceptions of "what goes on Inside the computer" for each of nine BASIC 

If 

statements. , 
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Users* Misconceptions of BASIC Programming Statements 
Following Hand^-On Experience 

Learning BASIC ^ 

This paper provides new Information concerning how beginning programmers 
learn BASIC. Suppose that you were going to teach a beginning programmer how 
to use a programming language such as BASIC. The typical Instructional ^ 
sequence Involves an orderly presentation of the elementary statements. For 
each statement, the. following Information Is generally presented: 
definition,' grammar, format specifications, example programs, and printouts. 
The apparent goal of most instructional sequences Is to teach the user to 
perform . The typical Instructional sequence occasionally Includes partial ' 
descrlptldhs of Internal processes of states In the computer — such as memory 
locations. Input stacks, etc. — but such descriptions are relatively rare and 
unsystematic. Thus, relatively little attention seems to be paid to the 
Instructional goal of teaching the user to understand . 
What is Learned 

The main focus of this paper concerns "what Is learned" when a beginning 
programmer is taught a computer programming language such as BASIC, following 
a typical Instruction sequence^. . * 

The outcome of learning — what is learned — can be viewed In two distinct 

ways: 

Learning BASIC Involves acquisition of new information . This idea 
asserts that learning a computer programming language is like learning any 
subject matter; the learner must acquire specific facts, skills, and rules. 
For example, the learner acquires new information such as foirmat rules for' 
when to use quotes in a PRINT statement or how to produce a conditional loop 
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using an IF statement. Acquisition of. specific information is generally the 
explicit objective of instruction. « 

Learning BASIC involves acquisition of a mental model s This idea 
asserts that, during the course of learning, a user develops a conception of 
the invisible actions and states that occur in the computer between input and 
output. For example, the learner acquires new mental models such as the idea 
of memory spaces for holding numbers as in a counter set LET statement. 
Acquisition of mental models is generally not an explicit objective of 
instruction. 

The distinction between the acquisition of new information and the 
acquisition of a mental model is just for practical purposes. If we conceive 
of the learner as an active, thinking individual, it becomes necessary to 
think of the two outcomes of learning as complements of each' other. Most 
instructional effort is directed towards the acquisition of new information. 
However, understanding of how mental models are acquired in the process of 
learning the new information might be useful for the instructor' as well a's 
the designer. One hypothesis is that basic training in the essential 
information coupled with the opportunity to practice at a computer terminal 
will naturally lead to the acquisition of "useful" mental models—models that 
can enhance understanding of the computer language. The present study aims 
to explpre this hypothesis. 
Mental Models for BASIC 

The present study explores the idea that learning of BASIC involves more 
than the acquisition of specific facts, rules, and skills • Beginning 
programmers also develop mental, models for the language in the process of 
learning the essentials of BASIC. Users' models, hoWever, may not be 
•accurate or useful ones. That is, users develop individual conceptions of 



Misconceptions of BASIC 

"what Is going on Inside the computer" as well as learn specific facts. 

Mayer (1979) has suggested a framework for describing the Internal 
transformations that occur for elemetttdry ^BASIC statements. 'In paYtit:ular, 
any BASI(/ statement can be conceptualized as a list of transactions. A 
transaction Is a simple statement 'asserting some action performed on some 
object, at some location In the computer. • For example, the following set of 
transactions describes what happens inside the coiAputer when the statement 
10 LET A = 0 is executed: 

,(1) Find the number in memory space A (ACTION: Find; OBJECT: Number; 

LOCATION: Memory). 
(2) Erase the number in memory space A (ACTION: Erase; OBJECT: 
Number; LOCATION: Memory). 
, (3) Find the number Indicated on the right of the equal sign (ACTION: 
Find; OBJECT: Number; LOCATION: Statement). 
(A) Write this number in memory space A^ CAGJION: Write; OBJECT: 

Number; LOCATION: Memory). 
(5)^Find the next statement in the program (ACTION: Find; OBJECT: 

Statement; LOCATION: Program). 
Experts and novices are likely to differ with respect to. their mental 
models for a programming language. For example, an expert programmer may 
have developed an accurate conception for a counter set LET, such As the one 
given above. That is, the expert knows that the value dn memory space A is 
replaced by 0. However, the novice may lack a coherent mental model or may 
possess incorrect, ones for BASIC statements. 

In a recent study, Mayer & Bayman (1981) asked novice and expert 
calculator users to predict the answer that would be displayed for a series 
of problems such as 2+3+ or 2+++«. Subjects' responses were matched to 

154 
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transactional expressions for each button press. Results indicated that 
subjects differed greatly in their conceptions of "what goes on inside the 
calculator" for each key press, with experts possessing more sophisticated 
conceptions than novices. Thus, although all subjects were able to use 
calculators to solve basic math problems — i.e. all users had acquired the 
basic information for performance — they differed greatly in how sophisticated 
a mental model they possessed. This work with calculators suggests that 
lea'i;nlng of the basic information about how to use a language does not 
guarantee that a user has also acquired a useful mental model for, the 
language. 

Moran (1981) suggests that the user develops a "conceptual model" of the 
system as he or she learns and uses it. He defines the user's conceptual 
model as the knowledge that organizes how the system works and how it can be 

r 

used to accompJLish tasks. How much training one needs to acquire a 
conceptual model has not been explored yet. Studying novicesV understanding 
of the programming statements constitutes the first step in addressing this 
issue. In this paper, we describe the novice programmers' conception of 
elementary BASIC statements, using a transaction analysis. There is yet no 
empirical evidence on novices T mental models of the statements of BASIC, at 
this level of detail. 

Method 

The goal of this study is to assess the nature of novice programmers' 
mental models for BASIC statements, following preliminary instruction in 
BASIC. In particular, this study assesses the transactions that each subject 
attributes to each of nine BASIC statements, following Instruction in how to 
use these statements. Some transactions are essential for understanding a 
statement. We will determine how many subject show evidence of having 
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acquired these essential transactions for each statement. Some transactions 
are Incorrect, such- as thinking that READ A Involves printing out a number. 
We will determine -how »any -subjects ^ow evidence of Incorrect txansactlons 
for' each statement. The final product of this study will be a frequency 
table for each statement, showing how many subjects appeared to possess each 
major transaction. 
Subjects 

The subjects were 30 undergraduateg at the University ot California,, 
Santa -Barbara who .had no prior knowledge about computer programming. 
Participation in the study was partial fulfillment of the requirements for 
the Introductory psychology course that they were taking. 
Materials and Apparatus 

Questionnaire , The questionnaire was an 8^x11 sheet of paper consisting 
of typed questions regarding: (1) the subject^s mathematics background; such 
as geometry, algebra, and calculus courses the individual had taken in high 
school and in college, (2) the subject's demographic characteristics; such as 
academic major, age, sex, CPA and SAT- Verbal and SAT-Math scores, and (3) the 
subject's computer programming background, such as whether the individual 
knew hoy to program a calculator and whether the individual had ever typed at 
a computer terminal, 

IBM Computer Programmer Aptitude Test , This pretest consisted of five 
8^11 inch sheets of paper based on a shortened version of the three-part IBM 
Computer Programmer Aptitude Sample Test (Luftlg, 1980), Part I (PAT-1) 
consisted of 25 number series problems, with a 5-mlnUte time limit. Part II 
(PAT-2) consisted of 18 picture analogies, with an 8-mlnute time limit. Part 
III (PAT-3) consisted of 12 arithmetic story problems, with an.8-minute time 
limit. 

I5u 
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Revised Mlhivesota Paper Form Board Test (Series AA) > ' This pretest 
consisted of 64 spatial vlsuallz^^rlon problems, with an 8-minute time limit. 

Lessons. Both Lesson I and Lesson II were composed from a 
self-Instruction, self-paced, mastery text called BASIC In Six Hours (Marcus, 
1980) that Is widely used In teaching BASIC In the Microcomputer Laboratory 
of the University of California, Santa Barbara. The text required that the 
user has hands-on access to an Apple-II Computer. The text did include 
.mastery tests at the end of each lesson but did not provide any conceptual 
models to explain the statements. 

Lesson I . This eleven page typewritten lesson covered statement 
execution in the immediate mode. The statements taught were the PRINT, LET, 
and IF-THEN statements.- There were exercises for doing simple arithmetic 
calculations, for having character strings printed on the display screen, and 
for assigning values to numerical and character string variable^ The use of 
the semicolons and commas for spacing on the screen, the use of the colon for 
statement stacking and the use of the command NEW were taught in conjunction 
with the statements. The lesson ended with a 10-minute self te^ that- 
covered ther essential information taught in the booklet. 

Lesson II . This fourteen page, typed lesson covered program 
preparation, adding or deleting statements in a program, obtaining the list 
of statements on the display screen, and how to run a program. Also the 
lesson covered the use of the GOTO statements, the IF statement, the 
numerical and character string INPUT statements, and the READ' and DATA 
statements. Several sample programs were Included, and there was a 10-minute 
self test at the end of the lesson. ^ 

Posttests . The study involved two posttests — verbal test and visual 

^ 2 
test— with the same nine statements tested in each. 
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Verbal posttest . This booklet contained 8^11 Inch, sheets of paper, 
with a single statement tested on each sheet. The statements In the fest 
, were (In order of presentation) i 
LET A » B + 1 

PRINT C » . 

LET D - 0 * 

PRINT "C" * • * 

IF A < B GOTO 99 

• INPUT A . . ■ • 

20 DATA 80, 90, 99 
30 READ A 
60 GOTO 30 

The Immediate mode statements were presented one on a sheet, with a carriage 
return (Indicated by <CR>) after each; the line numbered st.^cements were 
presented within a simple program. For both, subjects were^lnstructed to 
write. In plain English, the steps that the computer would carry out for each 
statement. They were Instructed to write each step on a separate l£h& of the 
test sheet. Subjects were Instructed to work on the statements in the given 
order and were not allowed to skip ahead or go back to the previous page. 

Visual Test . This test Involved the same nine statements as in the 
verbal test, presented in the same order and forma^ In addition, this test 
began with a three page typed introd%ctlon, which presented a visual diagram 
of the computer.' The diagram was based on earlier experiments (Mayer, 1981) 
and consisted of four parts— the display screen with keyboard, the memory, 
the input system, and the control system. An example of the diagram is given 
in Figure 1. For each of the nine statements, subjects were asked to 
indicate the state of each of the four components of the computer after a 
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particular statement was executed • For each statement there was a diagram 
that showed the state of the computer before the. statement was executed and 
another diagram that was left blank for the subject to fill In. 



Figure 1 about here 



Apparatus . Apparatus consisted of four Apple JI Plus computers, with 
48 K of memory, and with Amdek 12" Model 100 B/W monitors. 
Procedure 

The study consisted of six sessions. In the first session, subjects in 
groups of up to six filled out the questionnaire, received a brief 
description of the study, signed up for times for the next five sessions, and 
took the series of the four pretests. Tests were administered in the order, 
PAT-1, PAT-2, PAT-3 and Minnesota Paper ''ivdrm Board Test, with time limits of 
5, 8, 8, and 8 minutes, respectively. In ttra following three sessions each 




subject worked on the two lessons at I fts orher own pace. Subjects worked 
individually at the Microcomputer Laboratory,^ using Apple II computers in 
conjunction with the text. Following each lesson, subjects' performance on 
the self test was checked, they were asked to work on the questions they 
missed until they passed the test. In the final two sessions subjects took 
the post tests at their own pace, in groups of no more than five people at one 

# 

time. The verbal test was always given before the visual test. 

Results 

Scoring 

Each subject's^ protocol for each of the nine statements on the Verbal 
test was scored, by two judges. Each protocol was broken down into a list of 
transactions. A transaction expresses some action performed on some object 

, • ■ ^^"^ 
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at some locatiotv^ln tlie computer (s-*o Mayer, 1979) • For example, one 



subject's protocol for LET A = B + 1 was: 



1. 



Trints vhat is typed tin screen 



2. 



Understands what Xhe word LET means 



3- 



Solves the equation wit>. information given 



. 4. 



Gives a solution to problem 



These steps can be translated into the following transactions; - 
PRINT the equation on the screen. 



FIND the statement that was just entered. 



SOLVE the equation that was just entered. 

PRINT the solution on the screen. 

Disagreements between judges were rare and were settled by consensus. ^ 

Each subject's diagram for each of the nine statements in the visual^ 

3 

test was scored by one judge. The judge noted the specific contents of e^ch 
of the four memory spaces, the specific contents of the display screen, the 
contents of the input System, and the direction of the arrow in the control v 
system. (Note that executive control of order of line execution could not be 
determined in this test — so key aspects of IF and GOTO statements could not 
be scored.) Each subject;' s diagram was converted into a list of transactions 
by comjtaring the initial state of each component and the state of each 
component following statement execution as produced by the subject. For 
example, one subject fills in the components of the computer diagram for LET 
A ■= B + 1 as shown in Figure 2. As can^ be seen, the four memory spaces 
contain A « B + 1, 0, 21, 15, respectively (as compared to the Initial 
contents -of 0, 21, 15, 66, respectively), the display screen is intact * 
showing READY followed by the cursor^ the input system contains 5, 99, 6, and 
7 in each case and the control arrow points to WAIT (as compared tto RUN In 
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the Initial diagram). The transactions indicated by this subject are: 

For the^menory spaces Write A » B + 1 in memory space A, 

Move the numbers in each space to 
the next memory space. ^ 

For the control system Wait for the next statement to be entered 

from the keyboard. 



Figure 2 about here 



ERIC 



Por each of the nine statements a list of correct transactions was 
generated, based on ah earlier analysis of Mayer (197^). The correct 
transactions represent the events which are essential for describing the 
execution of the statement. For example, the correct transactions for 
LET A = B + 1 are as follows: 

1. Find the number in memory space A. 

2. Erase that number. 

3. Find the number in memory space B. 

4. Add one to that num}>er. 

5. Write the obtained value in memory space A. 

6. Find the next statement that is entered at the keyboard. 

The key transaction(s) for each statement were determined by two judges, 
based on the most characteristic eyent(s) in the statement. For example, the 
key transaction for LET A « B + 1 is, "Writ€ the obtained value in memory 
space A." ^ \ 

For each statement, a list of alternative transatSLtions was also 
generated, based on the subjects' answers. Some of the transactions \^xe ^ 
unnecessary, such as printing the statement on the screen. Some of the 

161 
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transactions were Incomplete, such as writing B + 1 In memory space A. 
Incomplete versions oji key trartsactlons, such as the previous example, were 
also Identified. Some of the transactlonewwere Incorrect , such as writing 
A « B + 1 in memory or printing the value of A on the "screen." The incorrect 
transactions represent students' misconceptions of ^ the events involved. for a 
statement. J) 
Freqaency of Misconceptions • " 

For each of the nine statements, a frequency table was generated by 
.tallying each subject's answer on the verbal test and on thfe visual test 
against the list of all possible correct, unnecessary, incomplete and 
Incorrect transactions. Tables 1 througTi 9 presents summaries of the data 
for each of the nine statements, respectively. Hence, each table, lists only 
the major transactions for a given statement, and shows the proportion of 
subjects who produced each transaction on the verbal test and on the visual' 
test. Proportions are reported separately for the verbal and visual tests. 
The missing proportions for a transaction on either test indicates that the 
transaction could not be detected from that test. For example, /'transfer of 
control from one line to another cannot be detected from the vlsual^tests. 
The transactions in the tables are stated in plain English. Key transactions 
are indicated by double asterisks (**) . 



^ Tables 1 through 9 about here 



> Note that op the verbal test subjects are asked to write the steps that 
the computer goes through in executing the programming statement. In 
contrast, on the visual test subjects are a^kedjTto fill in the foux-component 
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computer diagram (as shown in Figure 2) indicat?.ng the changes that occur due 
to the execution of the statement. The following expository discussion of 
results is based .pji the subjects^ answers in both of the tjssts but focufees on 

o • 

V 

the data from the verbal test. This is done because the verbal test allaws a 

c 

more detailed description of subjects' answers than the visual test, the 
verbal test was administered before the visual test, and the visual test did 
tiot provide information on transfer of control.' ^ * 

LET A-B-H . The major misconceptions for this statement can be grouped 
into three categories: (1) That the computer writes A=B+1 in memory or in . 
memory space A. Forty-seven percent of the subjects thought that tl^e 
computer stored the equation instead of the value obtained from B+1. 
(2) That the computer prints the equation or A or the value of A on the 
screen. Twenty-three percent of the subjects answered this way. (3) That 
the computer solves the equation A=B+1. Thirteen percent of the subjects 
believed that this is the case. 

Only 30% of the subjects* answers included the key transaction that the 
computer would store the value from B+1 for the variable A. 

. Table 1 gives the specif ic .proportions produced by subjects' ^inswers of 
the major incorrect and incomplete transactions as well as of the set of 
correct trans$ction^• 

(2) LET D-0 . The major misconceptions for thig statement can be grouped 

into three categories: (1) That the computer writes the equation in memory. 

\ 

Forty-seveo percent of the subjects opted for this idea. (2) That the 
computer solves the equation. Seven percent of the subjects answered this 
way. (3) That the computer prints the equation on the screen. Seven 
percent of the' subjects though*, this might happen. 

The key transaction that the computer writes the value zero in its 
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memory for the variable D is given "by 47 percent of the subjects. 

For more detailed description of the subjects' answers refer to Table 2 
which Includes proportlotis produced for an Incomplei^ transaction and for the 
set of correct transactions besides proportions for the major Incorrect 
transactions described above. 

(3) PRINT C > The major misconceptions for this statement can also be 
grouped into three categories: (1) That the computer ^Ints the letter C on 
the screen. Xhlrty-three percent of the subjects incorrectly answered this 
way. ,(2) That the computer prints either error or nothing on the screen* 
Seven percent of the subjects had this particular conception. (3) That the 
computer writes C in its memory. S^ven percent of the subjects held this 
idea. . ' 

The key transaction that the computer would print the value of the 
variable C on the screen was given by 40 percent of the subject population. 

See Table 3 for more detailed classification of the subjects' answers 
for this statement. 

(4) PRINT "C" . Three major misconceptions for this statement are: 
(1) That the computer prints the value of the variable C on the screen. 
Seven percent of the subjects answered this way. (2) That the computer 
writes C in its memory. Again 7 percent of the subjects thought this vas the 
case. (3) That the computer finds -the number in memory space C. Only one 
subject answered this way. 

Eighty-three percent of the subject population stated the key 
i;fansactlon that the computer would print C on the screen. 

See Table 4 for more detailed information on the subjects* answers for 
this statement. 

(5) I NPUT A . The major misconceptions for this statement are: (1) That 

, 0 
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the computer writes A in a data list or memory. Thirty percent of the 
subjects' answers fell in this category. (2) That the data or A will be 
printed on the screen. Only one subject held this conception. ^ 

There are three key transactions for the INPUT statement: (l> That the 
computer prints a question mark on the screen, Sev.en percent of the subject 
population stated this transaction. (2) That the computer waits for the 
yaiue of A and <CR> to be entered from the keyboard. Twenty-three percent of 
the subjects* answers Implied this conc^tlon. (3) That the computer stores 
the entered value In memory space A. Only one subject held this conception. 

See Table 5 for the proportions produced by subjects* answers for the 
major transactions for this statement. 

(6) IF A<B GOTO 99 . The major misconceptions can be grouped Into four 

categories: (1) That the computer prints number 99 or line 99 or an error on 

« 

the screen. Twenty percent of the subject population believed this was the 
case. (2) That the computer finds number 99 If A Is less than B. Thirteen 
percent of the subjects answered this way. (3) That the computer writes. A or 
B or A is less than B In memory. Ten percent of the subjects' answers fell 
in this category. (4) That the computer moves tb line 99 without any test of 
whether the condition A<B Is true I Ten percent of the subjects thought that 
t^ls waa the case. 

There are two key transactions for the conditional GOTO statement: (1) 
That execution would move toUine 99 In the program If the value of A Is less 
than the value of B. Sixty-three of the subjects held this conception. (2) 
That execution would continue with the next statement in the program If the 
value of A is not less than 'the value of B. Only 33 percent of the subject 
population stated this conception in their protocols. 

For more detailed list of the transactions and the corresponding 
proportions of subjects' answers see Table 6. 
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(7) 20 DATA 80, 90, 99 > The major misconception for -the DATA statement 
is 'that the computer prints the numbers on the screen. Thirteen percent of 
the subject population answered this way. 

Only 27 percent of the subjects gave the precise answer that the numbers 
80, 90, 99 would be put in memory or the input queue. 

See Table 7 for more detailed categorization of the' subjects' answers. 

(8) 30 READ A . The major misconceptions for the READ stiatement can be 
^roujped into' three categories: (1) That tjl:ie computer prints the value of A 
on the screen. Ten percent of the subjects believed in this idea. (2) That 
the computer writes A in memory. One subject stateci this conception. (3) 
That the computer waits for a value to be Entered from the keyboard. Again, 
only one subject held this wrong conception. 

Only 10 percent of the subject population answered that the first date 
value from the DATA statement would be written in memory space A. 

See Table 8 fpr more detailed description of the subjects' answers for 
the READ statement. 

(9) 60 GOTO 30 . The major misconceptions are: . (1) That the computer 
f indfe 30 if A is not equal to some number. Seven percent of the subjects 
answered this way. (2) That the computer prints line 30 on the screen. Only 
one subject gave this answer. * - 

There are two key transactions for the simple GOTO statement: (1) That 
the computer moves program execution to line 30 in the program. Sixty-seven 
percent of the subject population held this conception. (2) That the 
computer' continues with program execution from that line. Only 37 percent of 
the subjectsT answers implied that they had this idea. 

Table 9 lists these transactions and proportions corresponding to each 
transaction. 

- ' ^ - 
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Differences among statements . In order to makfe comparisons of subjects* 
conceptions among the nine statements, ekch subject's verbal protocol for 

V 

each statement was categorized as correct, incomplete, empty, or incorrect. 

The criteria for classifying protocols were: (1) correct — if the subject's 

protocol included the key transact ion (s) and no incorrect transactions, (2) 

incomplete — if the subject produced one or more incomplete vetrsions of the 
^ K * ^ ^ ^ * \ 

key transaction(s) and no incorrect transactions, (3) incorrecBc '-'lf tl 
subject produced one or more Incorrect transactions, and (4) empty — if the 
subject produced no key transacti^ons, no incomplete version of a correct 
transaction, and no incorrect transaction. Table 10 presents a summary of 

SI 

the proportion 'of users producing each type of conception for each of the 
nine BASIj^ statements. 

Table 10 presents the nine statements in order of difficulty based on 
proportion correct conceptions. As can be seen, PRINT "C" is the best 
comprehended statement — with 80% of the subjects expressing the correct 
conception—while INPUT A is the worst understood statement—with only one 
subject Indicating a. correct conception. It should also be noted that 
substantial numbers of subjects hold "empty" conceptions for the READ, DATA, 
INPUT, and PRINT C statements. 



Tablie 10 about here 



A corresponding summary table based on the visual test was also 
.prepared. Table 11 presents seven of the nine statements in order of 
difficulty based on proportion correct statements for this test. Data for 
the IF and GOTO statements has been excluded because tiie visual test did not 
allow for expressing transfer of control adequately. Although there seems to 
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be general similarity In the patterns of performance between the two tests » 
making specific comparisons between them Is not appropriate because of the 
differences in the nature of tests.^ Note that^the verbal test allowed * 
subjects to exhibit Incomplete and empty conceptions; in contrast > the visual 
test with its four-part computer diagram forced subjects to be specific* 



Table 11 about h^re» 



Pretests * An *add(itional analysis was performed in order to determine 
whether performance on th^ pretests was related, to development of the correct 
conceptions of the BASIC statements. First, correlations were run between 
each pretest score (MPFBT, PAT-1, PAT-2, and PAT-3) and the total number of 
correct conceptions for the nine^BASIC statements. Table 12 shows the 
coi?i:elation matrix, with significant r values marked with an asterisk(*). As 
can be seen, only PAT-2 — a spatial reasoning test — seems to be significantly 
related to haying correct conceptions for the BASIC statements (r .43, p < 
•02). Also, the two spatial reasoning tests, MPFBT and' Pat-2, show ft 
significant correlation (r « .49, p < .01) as well as PAT-2 and PAT-3 (r « 
•39, p < .05). In addition, a stepwise ^regression analysis was performed 
with each of the four pretest scores as the independent variables and the 
total number of correct conceptions of statements as the dependent variable • 
The resulting' equation selected only one variable — •PAT-2 — and accounted for 
18 percent of the variance in the de^^endent measuri. These analyses suggest 
that the PAT-t2 test was the qnly test that was related to understanding of 
the BASIC statements, although the- relationship is rather mild. 
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Table 12 about here 

; 

Conclusions and Recommendations 

i 

In this study we attempted to evaluate the mental models acquired by 
novices following several hours of learning and practice In BASIC. It Is 
Important to note that we focused on novices' learning of mental models 
rather than learning of specific Information. In order to evaluate mental 
models, we adopted a detailed level of analysis—transaction analysis (Mayer, 
'1979). This analysis has allowed us to Identify the misconceptions 'and 
missing cone*' ,lons in novices' understanding of the conceptual models 
underlying elementary BASIC statements. 

The results of the study Indicate that mastery of the information in the 
instructional booklet does not guarantee that users will acquire useful 
mental models for how the BASIC statements function. Although all of the 
subjects in this study were able to adequately answer the questions on the 
mastery test at the end of each lesson, only about one-third of the subjects 
were able to produce correct conceptual descriptions of an average statement 
such as PRINT C or LET A=B+1 and mqre than one-.half of the subjects generated 
incorrect or empt)i^conccfptual descriptions. 

Not surprisingly, the nature of the misconceptions are found jto differ 
from statement to statement; some statements being more difficult to grasp 
than others. The programming statements in order of difficulty (biased on the 
verbal test) can be listed as: INPUT A, 30 READ A. IF A<B GOTO 99, 
LET A « B 1, 20 DATA 80, 90, 99, 60 GOTO 30, .PRINT C, LET D - 0, and PRINT 
"C". However, this part of the results should not be taken as conclusive 
since we not only used a selective number of programming statements but 

16 J 



Misconceptions of BASIC 

presented them with a specific order In the testing sessions. One needs to 
compose an exhaustive list of the programming statements of BASIC and use 
different ptesentatlon orders to be able to niake conclusive ^remarks about the 
relative comprehenslbillty of each statement. 

In this paper 9 we analyzed beginning programmers* as a group so that we 
could focus on the most common misconceptions. There seems to be sufficient 
evidence for specific recommendations to be made for each statement incluiled 
in this study. 

1. Recommendations concerning INPUT . Subjects have difficulty in 
conceiving where the to-be-input data comes from and how it f s stored in » 
memory. Many subjects fall to understand the nature of executive 

control — i.e. that the computer, will "wait" for input from the keyboard. 
These learners need explicit training Concerning the role of input terminal » 
the wait-run control, and the memory spaces — including visual 
representations, verbal descriptions of key transactions (as listed in Table 
5), and role playing, by the learner. 

2. Recommendations concerning READ-DATA statements .' Subjects have 
difficulty in conceiving where the to-be-read data comes from and Low it is 
stored in memory. Subjects need explicit training concerning the data stack 
and the memory spaces — including visual represientations, verbal descriptions 
of the key transactions (as listed in Tables 7 and 8), and ^role playing by 
the learner.- ^ * 

3. Recommendations concerning the conditional GOTO and simple GOTO . 
Subjects* major difficulty with the GOTO is that they cannot conceive of what 
will happen next after program execution moves on to the desired line. Also, 
with the conditional GOTO, they s^em to have difficulty in conceiving what 
would happen next if the condition is false. Hence, beginners need training 
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at the terminal to observe execution control, explicit" training with verbal 
descriptions of the key transactions (as listed in Tables 6 and 9) and role 
playing to get a feel for execution control. 

4. Recommendations- concerning LET statements . Subjects seem to get , 
confused between solving an equation (I.e. treating the equal sign as an 
equalltjjl^and making an assignment. Those who seem to understand the 
assignment property in the statement still have difficulties In conceiving 
where to store the assigned values. Beginning learners need explicit 
training concerning the specific memory locations and under what conditions 
values stored in those locations get replaced. Visual representations, 
verbal descriptions consisting of the set of correct transactions (as listed 
in Tables 1 and 2), and role playing by the learner are ways to overcome 
beginners difficulties. 

5. Recommendations concerning the PRINT statements . Subjects seem to 
confuse the ^ function of PRINT C and PRINT "C'l. Also, subjects have 
difficulty in conceiving that these statements simply display on the screen 
what is asked to be printed; they incorrectly assume that the compute^^eeps 
a record of what is printed somewhere in its memory. Beginning programmers 
need explicit comparative training for the two types of PRINT statements. 
Training can be at the terminal so that learners observe the dif/'erehces in 
output; training can include visual representations for memory spaces showing 
no change after statement execution, verbal descriptions of the key 
transactions (as listed in Tables 3 and 4), and role playing by the learner. 

The present study has Jcocused on diagnosis of bugs in novices* mental 
models for BASIC statements. The specific diagnosis of what users do not 
know-^at the level of missing or incorrect transactions — allows us to develop 
individual instructional techniques f >r remediation. Thus, the next logical 
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step In this project Is to determine whether Instfuctlonar techniques can be 
develd'ped to correct users* mental models for BASIC. ^ 
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The test bookle&s used in this study Included additional questions that 

c 

were not analyzed, and do not Influence the present results. 
3 

Since scoring was literal, there was no need for an additional judge. 

4 

Some unnecessary. Incomplete and Incorrect transactions are excluded ► 
for purpose of clarity. However, detailed versions of these tables can be 
obtained from the authors upon request. 
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Table 1 

Proportion of "Users Who Produced Each Transactioa. for LET A=B+1 
Verbal Test Visual Test 




.43 



.23 



.13 



.33 



.20 
.03 
.03 
.03 
.30 

.27 



.17 



.17 



.17 



.20 



.33 



.60 



Incorrect Transactions 

Write A=B+1 in memory or in memory 

space A. 

Print A=B+1 or A or the value of A 
, on the screen. 
Solve the equation in the statement. 

Incomplete Transaction 

Write B+1 In memory space A. ' 

Set of Correct Transactions 
Find the number in memory space A. ^ 
Erase the number In memory space A. 
Find the number In memory space B. 
Add 1 to the number in memory space B. 
** Write the obtained value in memory 
space A. - 

Find (wait for) the next statement to 
be entered in the keyboard. 



Note, - Double asterlsk(**) indicates the kfey correct transaction. 

Dash ( ) indicates transaction could not be evaluated from the 

test. 
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Proportion 



~1 

Lon op Users 



Table 2 



sers Who Produced Each Transaction for LET D=0 



Verbal Test Visual Test 



* .47 
.07 
.07 



.13 



.07 
.03 
.47 
'.17 



.10 



.13 



.03 



.77. 



Incorrect Transactions 

Write the equation In memory. 

Solve the equation In the statement. 

Print- the' equation on the screen. 

IncompleC^e- Transaction 
Write D or 0 In memory. 

■ Set of Correct Transactions 

Find the number In memory space D, 
Erase the number ill memory space D. 

**Wrlte 0 In memory space D. 

Find the next statement to be entered 
In the keyboard. 



Note, — ^Double asterlsk(**) Indicates the key correct transaction. 

Dash (-i — ) Indicates transaction could not be evaluated from the 
test. * 
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Table 3 

Proportion of Users Who Produced Each Transaction fpr PRINT C 



Verbal Test Visual Test 



.33 



.07 



.07 



.10 



.17 



Incorrect Transactions' 

Print the letter C on the screen. 

Print either error or nothing on the 

screen. 

Write C In memory. 



.03 



.17 
.40 

.10 



.60 



I ncomplete Transaction 
Print 

Set of Correct Transactions 
Find the number In memory space C. 
** Print the number or zero on the 
screen. 

Find the next statement to be 
entered In the keyboard. 



Note. — Double asterisk (**) indicates the key correct transaction. 

Bash ) Indicates transaction could* not be evaluated from the 
test. 
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Table 4 

Proportion of Users Produced Each Transaction for PRINT "C' 



I Verbal Test Visual Test 



.07 
.07 
.03 



.07 



.07 

,83 

.io 



.17 
.03 



.47 



Incorrect Transactions 

Print the value of C on the screen 

Write C in memory 

Find the number in memory space C 

Incomplete Transaction 

Do not print the value of C on the 

screen. 

Set of Correct Transactions 
Find the letter C in quotes in the 
statement. 
**Prlnt C on the screen. 

Find the next statement to be 
entered in the keyboard. 



Note. — Double asterisk (**) indicates the key correct transaction. 

Dash ( ) indicates transaction could not be evaluated from the 

test. 
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Table 5 

Proportion of Users who Produced Each Transaction for INPUT A 



Verbal Test Visual Test 

.30 .17 
.03 ' .20 




^correct Transaction 



Write A in memory or data list. 
Print the data or A on the screen. 



.13 

.07 



.10 



Incomplete Transactions 

Wait for some data or number or A. 

Writie a number. 



.07 
.23 

.00 

.03 
.03 
.03 

.10 



,10 



Set of Correct Transactions 
** Print ? on the screen 
** Wait for the number and the <CR> to be 

entered from the keyboard. 

Find the number entered in the 

keyboard. 

Find the number in memory space A. 
Erase the number in memory space A. 
** Write the number just entered in 
memory space A. 

Find the next statement to be entered 

i 

in the keyboard. 



ERIC 



Note. - Double asterisk (**) indicates the key correct transaction. 

• ^- 

Dash ( ) indicates ttansaction could not be evaluated from the 

O .test. 
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Table 6 



Proportion of Users Who Produced Each Transaction for IF A<B GOTO 99 



Verbal Test Visual Test 



ERIC 



. .20 

.13 
.10 
.10 

.33 
.33 
.43 

.63 

.33 



.33 



.13 



.03 



\ 



Incorrect Transactions 

Print number 99, line 99, or error or the 
screen. 

If A is less than B, then find number 99 
Write A or B ot A is less than B in memory. 
Find line 99 in the program 

Set of Correct- Transactions 
•Find the number in memory space A. 
Find the number in memory space B. 
Test if the number in memory space A is 
less than the number in memory space B. 
** If the value of A is less than the value 
in B, then move to line 99 in the program. 
If the value of A Is not less than the 
value of B, th6n move on to the next 
statement in the program. 
Continue with the execution of' the program 
from there. ' ^ 



Note. — Double asterisk (**) Indicates the key Correct transaction., 

Dash* — ) Indicates transaction could not be evaluated from the' 
test.' 
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Table 7 

Proportion of Users Who Produced Each Transaction for 20 DATA 80, 90, 99 



Verbal Test Visual Test 



Incorrect Transaction 



.13 .13 : Print the numbers on the screen; 

Incomplete Transaction 
.27 .30 Put the data in memory space A or In memory 

Set of^ Correct Transactions 

.03 " Find the numbers in the statement 

,27 .60 **Put the numbers In memory or in the Input 

queue. 

,00 Find %he next statement In the program. 



Note. — Double asterisk (**) Indicates the key correct transaction. 

Dash ( ) indicates transaction could r not be evaluated from the 

test. ^ 
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Table 8 

Proportion of Users Who Produced Each Transaction for READ A 



Verbal Test Visual Test 



.10 
.03 
.03 



.07 

.07 
.00 
.10 

,10 



.07 
.07 



.13 



4 Incorrect Transactions 

Print the value of A on the -screen. 
Write A in memory'. , 
Walt for a value to be entered from 
the keyboard. 

Set of Correct Transtactlona 
Find the DATA statement In the 
program. 

Find the number in memory^ space A.. 
Erase the number in memory space A. 
^*Wrlte the first number from the DATA 
statement In memory space A. 
Find the tiext statement in the program 



Note. — Double asterisk (**) Indicates the key correct transaction. 

Dash ( ) Indicates transaction could not be evaluated from the 

test. ^ ' ' 
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Table 9 ^ 
Proportion of Users Who Produced Each Transaction for 60 GOTO 30 



Verbal Test Visual TesC 



.07 
.03 



.10 



Incorrect Transactions 

If . A does not equal to X or 99 f ind 30, 

Print J.ine 30 on the screen. 



.00 

•6? 
.37 



. Set of Correct Transactions 

Find the line number in the statement* 

**Move to line 30 in the program. 

**Continue with program execution from that 
line. 



Note. —Double asterisk (**) indicates the key correct transaction. 

Dash — )-»indicates transaction could not be evaluated from the 

. ft 

test. ' * - 
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Table 10 

Pro'portion of Users with Correct, Incomplete, Incorrect, and Empty 
Conceptions for the Nine BASIC Statements from the Verl^al Test 



Stateiaents 



Correct 



Conceptions 
Incomplete Incorrect 



Empty 



INPUT A .03 

30 READ A .10 

IF A < B GOTO 99 .27 

LET A = B + 1 ,27 

20 DATA 80, 90, 99 .27 

60 GOTO 30 .27 

PRINT C .33 

3 

LET D « 0 .43 

PRINT "C" .80 



.30 
.27 
.27 
.10 
.17 
.56 
.00 
.03 
.00 



.30 

.17 
.40 
.60 
.13 
.10 
.47 
.5^3 
.13 



.37 
.47 
.07 
.03 
.43 
.07 
.20 
.00 
.07 
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* Table 11 

Proportion of Users with Correct, Incomplete, Incorrect, and Empty 
Conceptions for the Seven BASIC Statements from the Visual Test 



Statements Correct 

INPUT A .00 

30 READ A . .13 

LET A >= B + 1 .30. 

PRINT "C" .37 

PRINT C .50 

LET D = 0 .60 

20 DATA 80, 90, 99 .60 



Conceptions ' 
Incomplete Incorrect 



.00 
.07 
.20 
.00 
.00 
.07 
.17 



.77 
.43 
.43 
.53 
.47 
.3Q 
.17 



Empty 

.23 
.37 
.07 
.10 
.03 
.03 
.07 
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Table 12 

The Correlation Matrix for Pretest Scores and the 
Number of Correct Conceptions for the Nine BASIC Statements 











* 




MPFBT 


PAT-1 


PAT-2 


PAT-3 Total Correct 


MPFBT 


1.00 






4 


PAT-1 


0.33 


1.00 






PAT-2 


0.49* 


0.22 


1.00 




PAT-3 


0.05 


-0.03 

f 


0.39* 


l.QD 


Total Correct 


0.24 


0.14 


0.43* 


0.04 1.00 



1 



Note. Asterisk (*) indicates p < .05. 
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Figure Captions 

. ,[ 

Figure 1. pia£ram of the computer used in the visual test. 
Figure 2. One subject's changes of the contents of the components of 
the computer diagram for LET A»B+1. ^ 
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The display screen and 
keyboard: 





The memory: 



6 
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w 
u 




The control system: 
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Initial state: 











a 


2f 


75 


66 




SubJ ect fills in the cc«nt>on6nt s as ; 













0 


2/ 


1^ 



WAIT 
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CHAPTER 5 



* 



EFFECTS OF INSTRUCTION ON' REMEDIATION OF BUGS AND 
PROBLEM SOLVING FOR CALCULATOR LANGUAGE 



/. 



^ 



Note ' " * • 

" "» 

This article^ has beeu submitted for publication as followa: ^ 

Bajnnan, P.» 4 Mayer, R. 5. Instructional manipulation of users' mental 
models for calculators'. International Journal of Man Machine 
Studies; under review* 



19 



u 



\ 

^ V, 168 

Abstract 

A mental model for a calculator or computer referfe to the user's conception 
of the "invisible" information processing states and transformations that^ • * 
»occur between input and ^output* This paper explores the following ideas 
concerning users' mental models for electronic dalculators: (l) users differ 
, greatly in their mental models in spite of similar "hand«-on" experience, 

(2) users ofteix tend to develop either impoverished or incorrect models,' 

(3) useprs, can be encouraged' to develop more useful mental models ^through 
instructional intervention. ^ 

, Key Words and Phrases: calcvilator, instruction, learning, psychology 

CR Categories: 1.50, 2.12, If. 20 . . 
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I> Mental Models 

Work on htunan factors related to computers—or what Moran (I98I) calls 
"an applied psychology of the user" — has "been drawing increased attention. 
Within' the past year, special issues of the International Journal of Man- 
Machine Studies (Volume 15, Number 1, July I98I) .and of Computing Surveys 
(Volume 13, Number 1, March I98I) have been devoted entirely to this topic, 
and the TT^Ti! Computer Society recently commissioned a tutorial text. Human - 
Factors in Software Development > edited by Bill Curtis (I98I). An emerging 
theme from human factors- work concerns the important role of w;hat Yoiing (l98la, 
1981b) calls the user's "mental model" or what Moran (I98I) calls the "user's 
conceptual mo.del" of the to-be-leamed system. 

In the course of learning to use a new computer language or a new calcu- 
lator language, the user may develop a "mental model" of the machine. A 
mental model is a iSietaphor consisting of the components and the operating 
rules of the machine. A mental model of a calcvilator allows the user to 
conceive of "invisible" information processing states (such as contents of 
internal registers) and "invisible" transformations (such as moving the contents 
of one register into another) that occur between input and output. Mental 
models are particularly important when users will have to guess the current 
state of the machine, or have to make inferences about how a given sequence 
of commands produces a certain output. 

Examples of mental models have been proposed for calculator langiiages 
(Mayer & Bayman, I98I; Yotmg, 198la, 198lb), text editing languages (Card, 
Moran & Newell, I98O; Moran, I98I), file management languages (Mayer, I98O, 
198I; Reisner, I98I), LOGO (DuBoulay & O'Shea, 1978; DuBoulay, O'Shea i Monk, 
1981), and BASIC (Mayer, 1979, 1981). For exaxaple^ Young (I98la, 198lb) suggests 
a siinple "register model" for a four function calcxaator. The components 
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are: a keyboard for entering niunters, operators, and equals keys; a display 
for shpving one nxomter; an internal niomter register for holding one numter;-^ 
and an internal operator display for holding one operator. The operating 
. rules include the fol4>wing: the first niomher that is entered is stored in 
'the internal nxomber register; for example, the sequence 2+3= vill result in 
2 being stored in the internal number register. The last operator symhol 
that is entered will "be stored in the internal operator register; for example, 
the sequence 2-+3= will have tHe same effect as 2+3= "because for "both sequences 
» + • is the last operator entered. The last niomher that is entered is stored 
" in the display; for example, 2+3 will result in 3 heing shown in the display. 
When' the. equkl key is pressed, or when an operator key^ is pressed after an 
expression has been entered, the operation will he c^ried and the result 
shown in the display; for example, 2+3= will have the .same effect on the 
display as 2+3+. If an equal key is pressed ^after an operator key, the oper- 
ation will he carried out using the number in the internal number register 
as both the first and second operand; for example, 2+= would resialt in the 
display of U, since '2' is added to itself. This relatively simple model— • 
consisting of a few components and operating rules—is atle to generate answers 
for any sequence of key presses. 
2. Useful Mental Models 

Users' mental models may vary along several dimensions, (l) Usefulness. 
A mental model is usei\a to the extent that it supports sophistxcffted per- 
formance hy the user. (2) Completeness . A mental model is ,complete to the 
extent that all details concerning the components ajid operating rules are 
spelled out . (3) Veridicaltty . A mental model is veridical to the extent 
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that the components match the actual physical design of the machine and the 
operating rules match the machine language. The present study is based on 
the idea that the first dimension , is important for users whi;Le the stecond 
and third are important for electronics experts. 

Users who acquire mental models through "hands-on" experience are apt 
to develop models that are sufficient for simple computation, but that lack 
sophistication required for complex mathematical work. Similarly, 'most calcu- 
lator manuals focus on procedures for solving simple computational problems, 
but make no attempt to help the user generate useful conceptual models. 
^In other words, learning to use a calcula-^or through hands-on experience or 
manual reading may not produce^ "useful" mental models. Although a user's 
mental model may be internally self-consistent and appropriate for standard 
computations, different users may develop quite different models, varying 
in usefulness, (l) Incorrect performance . A user may possess a model that 
is consistent with simple expressions of the f<ipa», 2+3=, but which generates 
incorrect perf^jrmance when extrapolated beyond simple expressions. For example, 
Mayer* & Bayman (198I) found that^" many users assume that an expression is evaluated 
as soon as a number key is pressed, such as predicting that 2+3 results in 
5 appearing in the display. (2) Im(poverished performance . A user may possess 
a model that is consistent with simple expressions of the form, 2+3=, b\^ which 
does not always generate correct performance when extrapolated beyond simple 
expressions. For example, Mayer & Baymaja (1981) fo\and that many users assume 
that an expression is evaluated only when an equfiULs key is pressed. jSjL 
Sophisticated performance . A user may possess a model that is consistent 
with simple expressions and which also can be usefully extrapolated to complex 
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expressions. For example, Mayer & Bayman (I98I) found that som^ users assume 
that an expression Is evaluated whenever an^ operator key (or equals key) is 
pressed, such as predicting that 2+3+ results in 5 "being displfiyed; 

« 

3^ .Instructional Study 

The goal of the present study is to determine whether useful models for 
calcxilators can be -explicitly taught to users. In this ^tudy some users 
are given modest instructional intervention, namely, introduction of a diagram 
showing some internal registers. For some users (line group) the diagram 
presents the internal registers in a line, alternating between "number" and 
"operator" registers; for other users (stack grii^up) the diagram presents the 
internal registers as a stack of "number" registers yith a single "operator" 
register to the side; for other users (no model group) no description of 
internal registers is given. If the introduction of the diagrams encourages 
users to build more useful mental models, then users who are given the diagrams 
should show more "sophisticated" perforaance and less "incorrect" pr "impover- 
ished" performance. .^.--^ 

3.1 Method 
♦ 

Subjects and design . The subjects were ^2 students recruited from Intro- 
ductory Psychology courses at the University of California, Santa Barbara. 
Subjects had no previous experience yith computer programming, but all were 
casual users of cdlculators. Twenty-four subjects served in the line group, 
2U. served in the stack group, and 2h served in the* np model group. 

Materials . The materials consisted of a pre-experimental questionnaire, 
three instruction booklets, and three problem booklets. 

The pre-experimental q^uestionnaire was an 8 1/2 x 11 inch sheet of paper 
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with typed questions. The questionnaire asked about the subject's age, SAT 
scores, sex, maljor, year in school, previous mathematicte- courses, experience . 
with computer programming, ownership of calculators, and experience with 
calculators. , ' • - * 

The instruction booklet for the line group contained tliree 8 1/2 x 11 

/ " . • 

inch sheets of typed paper. The first page presented a diagram of the key- 
board and display of a typical fo\ar function calculator (see Figure l), and 
described the digit and operator keys that were relevant to the experiment. 
The second' page presented a diagram arid description of the internal registers 
based on a line model (see Figure' 2). The internal registers were represented 
as a rov of alternating lines and boxes, such that a numbe;r dould be held in 
each line and an operator in each box. The display was drawn as a rectangle, 
and was described 9,s that part of the calculator where the answer is shown., 
The third page provided instructions for the task and gave four sample problem^ 
Hach sample problem consisted of a sequence of key presses, such as 3 + 2, , 
and was accompanied by a diagram of the internal line register and 'display 
for subjects to fill in; ^Sub^ects were told that their Job was to fill in , 
what numbera and symbols would be in the internal' registers and display after 
the final key press; , 

The instruction booklet for the stack group also consisted of three 
typed 8 1/2 x 11 inch sheets. The first page was identical to the first 
page of the lipe group's booklet. The second page presented a diagram and 
description of the internal registers based on a stack model- (see Figure 3). 

e internal registers w6re represented as a column of rectangles labeled 
x-register, y-register, and operatoi^register. The description pointed' out ^ 



f 

that the x-register, y- register, and display could hold niimbers and that 
numbers covad be transferred f rom x>ne to the other, while the operator register 
could hold an operator* symbol. As with the line model, the display was repre- 
sented as a separate rectangle and as the only visible part of the calculator. 
The fineil page provided the same task instructions and sample problems as 
in the line group's booklet, except that the diagram accompanying each samfple 
problem was of the stack model. 



Figures 1, 2 and 3 About Here 



The instructional booklet for the no moael group consisted of two typed ^ 
^ 8 1/2 X 11 inch jsheets of paper. The first page was identical to the first 
page for the other booklets; the second page was identical to the last page 
of the other ^bdoklets except that only a display rectangle accompanied each 
sample problem and the subject'^s task was to write down what would be in the 
display after the last key press. 

Three problem booklets were constructed using 8 1/2 x 11 inch sheets 
of paper. Each sheet, contained four to seven problems, witfi each problem 
" consisting of one to seven key strokes. Each key strdke was either a digit 
(2, 3, or 7), an operation (»- or x) , or an equals (=). For the line group, 
the line model and display were given to the right of each problem; for the 
stack group, the stack model and display were given to the right of each ^ 
problem; for the no model group, the display was given to the right .of each^ 
problem. Examples are given in Figure k. All' booklets contained the same 
25 problems in common/ Examples are given in Table 1. As caiH)e seen, some 
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problems involve standard sequences of key presses (standard problems or 
chain problems) while others * involve sequences- that violate the grazmnatical 
rules of arithmetic (non-standard problems T. 

r 

» 

I « 

Figure U and Table 1 About Here 

3>2 Results 

The number or characters that a subject thought vould be in 
the display was recorded for each of the 25 problems for each subject. Scoring 
was then broken down into three subsets of the data: (l) Answers to problems 

I through 10 (a^ partially shown in Table 1) were used to determine the sub- 
lect's conception of when an exjpression is evaluated. (2) Answers to problems! 

II through 23 (as partially shown in Table l) were used to determine the 
subject's conception of how to evaluate a nonstandard sequence. (3) Answers ^ 
to problems 2h and 25 (as shown in Table l) were used to determine the subject's 
conception of the order of execution of operations within a chain. 

' For problems 1 through 10, earlier studies by Mayer Sb Bayman (1981) suggest 
three types of strategies concerning whe?n to evaluate an expression. 

- (1) Number strategy . According to this strategy, an expression is 
evaluated as soon as a number key is pressed. Thus, for the problems, 2, 
2+ , 2+3, and 2+3+, the answers are, respectively, 2, 2, 5, 5* This strategy 
can be considered "incorrect" because it does not allow for two digit npbers. 

(2) Equals stfategy. According to this strategy, an expression is 
evaluated only when th6 equals key is pressed, ^fhus, for the problems, 2, 
2+, 2+3, and 2+3+, the answers are, respectively, 2, 2, 3, 3. This strategy 

ft 
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can te considered "impoverished" because it is limited problems in standard 
algebraic rotation as taught, in teit books. . 

(3) Operator strategy > According to this strategy, an expression is 
evaluated when either an operator key (such as + or x) .or the equals key 
'(=) is pressed. a?hus, for the prgblems, 2, 2+^ 2+3, and 2+3+, the answers 
are, respectively, 2, 2, 3, 5. This strategy can he considered sophisticated 

hecause it goes heyond the normal jalgehraic notation, and attritutes new 

s ■ „ •* 

actions to the calculator. The right panel of Tatle 1 shows the predicted 

■' •• ' " • ■ . 

answers for several of the first 10 prohlems for each of the three general 

strategies, " . • 

■ 

For prohlems 11 through 23, earlier studies hy Mayer & Bayman (1981) -suggest 
three types of strategies concerning how to evaluate a nonstandard sequence. 

(1) Ignore strategy. According to this strategy^ any sequence of symhols 
that violate the grammasp of arithmetic is ignored. For the prohlems, 2+=+=+= 
and 2++3=, the answers; would he, respectively, 2 and 5. This strategy can 

he considered "impoverished" hecause it mirrors the standard algebraic notation 
as taught in textbooks. • ' % 

(2) Reset strategy . . According to this strategy, any sequence that 
violates the grammar of arithmetic will he detected and the display will 
produce an error -toes sage. For'the prohlems, 2+=+=+= and 2++3=, the answers 
would he, 0 or E or 2 for the first ajid 0, E or 3 for the second. This strategy 
is more sophisticated than the preceeding 'one because it assumes Ijhat the 
calculator can detect and point out errors. 

(3) Increment strategy . According to this strategy, any sequence that 
violates the granmiar of arithmetic J.s subject- to incrementation. For the 



1^ 



J.* ^ . , 

problems, 2+=+=+= and 2++3=, the answers covld be 6 or 8 or l6 tar the first 
and 7 for the second. Tljis strategy is "sophisticated" because .it ^ allows 
for an actively operating calciaator that can go beyond the rules of;^tandard 
textbook notation; the increment strategy suggests that the calculator inserts 
a nmber between- two consecutive n6n-nimieric symbols. The right panel^ of % 
Table 1 sliows tl^e predicted answers for several of the problems 11 through 
23 for each, of the three gener^al strategies. . ^ 

For problems 2k and 25, earlier studies by Mayer & Bayman (l9^l) suggest 
three types of strategies concerning the order of execution of operations . 

in a chain. ^ 

(1) Incorrect strategies ! Several incofrect strategies are possible, 
including performing the chain from right to left (i.e., w^th answers of 
23 and 20, respectively), or performing additions "before miiltiplications 
(i.e.-, with answers of k2 and 20, respectively). T]^ese strategies are incorrect 
because they violate standard algebraic notation and' rule? as taught in text- 

books. , 

(2) Left-to-right- strategy . According to this strategy, the operations 

are performed in order beginning on the left (i.-e., with answers of 35 ^d 
13, respectively). This is an "impoverished" strategy because it is based 

7 ' * ^ • 

solely on a knowledge of simple arithmetic notation as taiigiht in textbooks. 

(3) Mult iplication-bef 6re-addition strategy . A more sophisticated 
strategy, that assumes active processing by tiie calculator, is that multipli- 
cation operations are performed before additions (iie. , 23 and 13, respectively). 
The right panels of Table 1 show the pre'dioted answers for problans and^ 

25 for each of the three . general strategies. 



2Uv 



.4 



For each subject, 'a tally wa^ made^of the, niimber of answers for problems 

1 thrdugh 10 that malfched an^wer^ for each of the three strategies for when 

to evaluate" an expression. The 'best ^fitting 'evaluation strategy for each . 

subject W3.S determined "by selecting' the strategy that most often produced 

e^swers matching those given "by subjects on problems 1 through 10. Table 2 

shows the proportion or subjects in each group who were classified as using 

number, equals, and operator strategies. As^ can be seen, -^here is a "trend 

in which line subjects and stack subjects are morp likely. to evaluate f6r . 

an operator (i'.e.j the sophisticated strategy), tliarx the control ^oufi« 

A'chi-square test was conducted on the ,4ata, witl^ the iiumher and equals , ; / 

strategies collapsed dntt) on6 ceil. The cHi-square te^t revbaled that the 

- > ■ , ■ - ^ i ' -'^ 

three e^ou^s differed significantly wi-th- respect to the proportion of suhjects using 

' - " ■ ' 2 • ^ • . 

a 'sophisticated strategy, . x = 9.01, df = 2, £<.02. 
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, Tahle 2 Ahout Here » ' „■ 

)$: ' . ' 

fW __ ^. 

■\ 1 ' ft 

For each suhject, i tally was made of the nunib6r of answers fo^jfj^fotlems ^ 

. 11 through 23 that matched answers for each of the three strategies fc^thoy. - 

to evaluate a non-standard expression. The test .fitting stratejgy was deter- 

mined for each suhject as described ahove, aifd Tahl? 3 gives the proportion 
' » ' • ' . ' V ■ . 

<* bf subjects fronj each group that were classified a# using .the ignore, reset,- 

r • • 

and increment strategies. As can he seen, the line group and, to' som^. extent, • 

■> ^ .' " • • ■■ . 

the stack group, tend to he more likely than the control group to use sophis- 
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ticated strategies such as increment or reset. A chi-square test was con- 
ducted on the data in Tahle 3, with the reset and increment .strategies . . 
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collapsed into one cell. The chi-square test revealed that the three groups^ 
differed significantly with' respect .to th« pl-oportion of subjects using ' ; 
sophisticated strategiies, x = 6i20^ df = '2,;£< .05- 



Tables 3 About Here 

V* 

For each Subject, a tally was made of the number of answers for problems 
2k and 25- corresponding to each of iiie three chain strategies. The best 
fitting strategy was detennined for each subject asc^escribed above, and 
Table k siimarizes- the ^ proportion of subjects from each group who were clas- 
sified as using incorrept, left-to-right, and mult iplyr be fore-add strategies.- 
As can be seen in the table, there are no major differences among the groups. 
One j-eaaon for the failure to identify any differences antor^g the groupp may . 
be that only two problenis were u§^ed on the test. . ^ 



Table h About Here 
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U. Conclusi ons and Recommend9.tions , " 

This study provides information concerning three iisijor conclusions.^ 
1. Users differ gratatly in their. me'ntal models despite similar hands-on 
experiehce . • For e^^ample , in the control grouj? ahout a third used the number 
Strategy, about a third used the equals strategy, and about a third used the 
operator strategy for when \o evaluate expressions. Presumably, all the 
subjects 'were ab\e %o competently use their calculators for basic arithmetic 
computations, but they acquired quite different conceptions o£ how calculator 
language operates. v ' - 



i 

' J- 
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2> Users often develop impoverished or incorreQt models . For exfample, 
a third of the control subjects and Q% of the model subjects tend to exhibit 
an incorrect strategy (i^e., "number strategy") for when to evaluate an ex- 
pression. Thus, there is adequate reason to believe that subjects who l^arn 
solely by "hands-on experience" may develop mental models that either are 
quite limited or otitright vrong. 

3. Users can be encouraged to develop more useful mental model s through 
Instructional intervention . Mayer & Bayman (1981) found subjects who Jiad pro- - 
gramming experience '(i*e. ,' experts ) were more likely to use sophisticated 
strategies for when to evaluate an expression (i.e., operator strategy) and 
how to evaluate a non-standard expression (i.e., increment and reset strategies) 
" than non-programmers (i.e., novices). In the present study, the tendency to' 
act like experts was enhanced by teaching casual users to make use of simple 
models of the internal registers of tlie calculator. Thus, the present study 
demonstrates that mental models can ije expilcltlrlJauS^ users of electronic 
computing, devices, and that these models can enhance the level of sophistica- 
tij)n of user performance. Of course, it is not argued that such an instructional 
interve?ition as" ■ used in^ this, study would be most helpful for alt nonexpert- 
users; neither is it argued that any one mental model would work for all 
individuals.' Further research is necessary to reveal "good mental models" 
tiiat would help the majoiffty of individuals to have a better grasp of the 
operation of the electronic device that they work with. 

This study suggests recoWendations for how to provide and evaluate in- 
struction^ for electronic computing devices such as calculators and computers. 
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1. Provide the novice user vith a simple and useful model -of the internal 
components and operating rules . Some models may be more effective than others. 
For example, the line model may he more useful for calculators using alge- 
braic logic while the stack model may he more useful for calculators using 
Reverse Polisli Notation (RPN). 

2. Encotirage the user to relate "hands-on experience" or "manual in- 
structions"' to the model . Hands-on experience does not guarantee that the , 
user will "understand" what he or she is doing. Thus the user ^hould he 
challenged to predict what is going on "inside" the device during computations. 
Similarly, asking a user to learn solely from a maiual may also produce un- 
sophisticated mental models, since manuals generally do not discuss internal 
components and processes. When manuals fail to provide adequate models, 
instructors should -provide the models and encourage users to "translate". 

the manual with respect to the models. 

v. . 3. The usefulness of a model should he e-Q^aluated . There is no guarantee 
that all models will increase the' sophistication of user performance. One 
yardstick is to compare novices who use various aodels (or no model) against 
experts. Models that increase the tendency to act like experts in a domain 
are useful; those that do not make users act like ejcperts are not useful ^ ,. 
and should "be replaced. 
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Table 1 

Examples of Problems Used in the Study 



Standard Expressions (Problems 1-10) 
2 3 

2 + 3 + . 
2 + 3 '+ 7 

Non-Standeird Expressions (ProTjlemy 11-23) 

2 + + = 
2 X - 
• 2 X X =_ 
2 + X 3 = 

Chain Expressions (Problems 2l;-'25) 

V 2*»v> 2 +3x7 

2 X 3 ' •»• 7 
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Answer for • 
Number Strategy. 
5 
5 

• 12 
Answer for 
Ipriore Strate^ 
2 

2 • 

^ 2 

2 

5 or 
Answer for 
Left Strategy 
35 
13 



Answer for 
Eqtials Strategy 

3 
7 

Answer f&t 
Reset Strategy 
0 
0 
0 
0 

3 or 0 

Answer for 

mtlply Stratepy 

^ 23 
f 

Z 3,3 



Answer for 
Operator Stratetgr 
3 
5 
7 

^ Answer for 
Increment Strategy 
k 

1» or 6 or 8 

U op 8 or l6 
12 or 15 
Answer for 
Other Strategy 
(varies) 

(varies) . S 

1 
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Table 2 • . 

r 

Nmter of Subjects From Each Treatment tooup Who Were Classified 
as Ufing a Number, Equals, or Operation Stratdgy for When to Eveauate 

an Expression Problems 1 to 10 



GroTXp 

Line Group (n=2U) » 
Stack tJroup (n=2U) 
Control (n=2U) 



Evaluation Strategy 
Number Equals Operator 



6 
10 
9 



l6 
12 
7 



Note. - Chi-square test indicrfes significant difference 
among^ strategies us^ by the ^;hree groups, £< *62. 
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Table 3 ^ ^ 

Number of Subjects From Each Treatment Group Who Were Classified 
tising an Ignore, Reset, or Increment Strategy for How to Evaluate 
a Nonstandard Expression . — Problems 11 to 23 



Group 

Line Group (n=2U) 
Stack Group (n=2U) 
Control (n=2U) 



Evaluation Strategy 
Ignore Reset Increment 



10 ■ 6 
15 3 
18 0 



8 
6 

6 



Note.. - C!hi-square test indicates ^gnificarjt differences 
among strategies used by the three groups,, £<. 05- 



21 



188 



Table h 

Number of Subjects From Each Treatment Oroup Who Were Classified 
as Using a Left-to-Right , Multiply-'betore-Add, or Other Strategy 
for How tO' Evalviate a Chain of Expressions — Problems 2h to 25 



Group 

Line Group (n=2l|) 
Stack Group (n=2li) 
Control in=Z^"\ 



Evaluation Strategy 



Left-to-Right Mult iply-hef or e-Add Other 



X8 
17 
17 



2 

i 



k 
6 

k 



Note. - No significant differences among groups in strategy usage. 
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Figure Captions 
Figure !•> The Foixr Function Calculator 

Figure 2. The Line Model ' ' ' . 

Figure 3- The Stack MDdel 

Figure k. Exainples of Problems for Line, Stack and No Model Groups 
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Figure 1. The Pour Function Calculator 

* 

* 

In thi^ experiment, you should pretend that you have just been given a 
simple handheld calculator. Your calculator looks like the one shown below. 
Notice that it has. 10 digit keys (labeled 0 through 9); 4 fjjnction keys for 
. addition (+)» subtraction (--), multiplication (x), and divisdon (f)} 1 equals 
key (=); and* a digital display which presents numbers. You can presS* k^ys in 
a certain order, and the answer will appear in the display. 




Figure 2. The Line Model 
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As ypu can see, the diagram on the previous page shows what the calculatoi^^ 
Ipoks like on the outside. We also wapt you to think about what goes on inside 
the calculator as you press each key. There are twa internal parts of the 
calculator that ve^jK^at-you to consider,' as shm^n below. These are the dj.splay~ 
register (which is. also seen on the outside) , and the main-register that contains, 
lines ^nd boxes for the numbers and the 'operator symbols to be registered respec- 
tively.- Each line may hol<J^ one number up to 8 dibits in length. Each box may- • 
hold one operator symbol (such as +, x, or v). ''It is possible to copy, a 
number frpm phe display-register to one of the slots in the main-register. 
iYou must fill-in the leftmost slot or box first and work from left „to right 
within the iialn-register. Only the display-register is visible during calcula- 
tions atid thus this Us where the answer will be shoxra. ' ^ 



•[ 



3 



Figure's/ Thp Stack Model 
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^As you can see, the diagram on the :previous page ^hows what the calculator 
looks like on the -outside. , We also want yoii t\ think ab6ut what goes on inside 
the calculator as press each key. There are four internal parts jif the 
calculator that we want you to consider, as sha\/n below. The four parts are 
the display-register (which is also seen on the outside) the X-register, the " 
Y-register, and the'. operator-register . ^ ' ' .: 

The operator-register can hrbld one operatoK^ symbol (such as x, or t) 

'at a time. If you press , a function key, the previous operator symbol sitting 
in this register gets erased and the new one you just pressed goes in the register. 

The three other registers, the display-register, the X-register, atid th^ 
Y-register can hold numbers up to .8 digit* in \ength^ . These registers can carry * 
one number at a time. If there i$ already a number in thfe register and if you 
put a ;new number i^ the sam6 register, the old one sitting there will be er-sed. . 
Hov;ever, you may copy a number from- one register to another (such as froni Liie 
display-register to the X-register or f roin^ the X-registeV td the Y-register). 
Among the registers only the -display-register is visibl^ during calculations 
and thus -this is where the answer will be shm^n. • 



Y 
X 



mi:. 



Figure Examples of Problems for Line, Stack, and- No Mqdel 



Line Group 



2 + + = 



.o_o_D_D_n_a 



[ 



OlSfUAY 



Stack. Group 



2 + + = 



1>ISPUAY 



No Model Group 



2 + + = 




' CHAPTER 6, 
EFFECTS OF INSTRUCTION QN s REMEDIATION OF BUGS AND 
PROBLEM SOLVING FOR THE BASIC LANGUAGE 
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Note 



This researgh will submitted for publication as follows: 

Bayman, P., & Mayer, R. E. Effects of instructional method on users* 
mental models for BASIC programming statements.' Journal of 
Educational Psychology , in preparation. » 
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The main purpose of the present study Is to determine whether the format, 
of infitrnction for a computer programming language can influence the learner's, 
mental model of the computer. The user's mental mO^el refers to the user's 

conception of what happens inside the cpmjputer between Input and output. 

<? 

Previous research (Bajnnan & Mayer, in press) found that when college 
students study BASIC through a standard manual coupled With hands-on 
experience, they develop several fundamental misconceptions regarding the 
meaning of programming statements. For example, about one half of the users 
thought that LET A = B + 1 resulted in the computer storing an equation in 
memory. In addition, about one-third thought that PRINT C meant that the 
computer would print the letter C on the screen, and about 10% thought that 
READ A meant that the computer would print the value of A on the screen. It 
should be pointed out that subjects were able to solve the problems in the 
manual, and were able to successfully engage in hknds-on contact with the 
computer. However, in spite of mastering the problems in the manual, most 
users developed mental models that were either incorrect or incomplete. 

Developing an appropriate mental model in conjunction with one s first 
prpgranmlng language may be critical to one's success in transfer and in usin^ 
the language creatively. , Hence, the present study is aimed at testing whether 
some instructional methods can help leamets to develop useful mental 
models-- i.e. , the present study examines whether mental models can be taught 
as part of normal instruction. 

Several researchers have recommended that novices should *leam their 
first language in a way that allows them to spontaneously build a mental model 
of the system (Carroll & Thomas, 1982; Moran, 1981; Young,* 1981) . Mayer 
(1981) has also summarized research concerning the rol6 of explicit mental 



lodels in learning computer programming. 
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A task analysis of the BASIC language (Mayer, 1979) reveals that each 
Statement can be described as a list of '"transactions" — l.e,, an action 
perforraect on some object In some . computer location. For example, the 
following transactions are Involved In the statement LET A = 0: 

1. Find the number In memory space A, 

2. Erase the number in memory space A. 

3. Find the number to the right of the equals sign In the 
statement. 

># 4. Write this number in memory space A. 

5. Find the next statement ln^#he program. 
Hence, through a set of transactions, we can clearly specify this actions that 
occur within the computer as any statement Is executed. The present st\idy 
attempts to explicitly teacA the transactions Involved In each statement, Ih 
order to determine - whether such explicit Instruction leads to less 
misconceptions and better criterion performance. * ^^^^^^.^-^ 
Method _ * . 

Subjects . The subjects were 95 college students at the University of 
California, Santa Barbara. The subjects had no previous experience or 
knowledge about computer programming. 

Design . The subjects were randomly distributed Into five groups: 
(I) Standard Group, In which the subjects studied BASIC from a standard 
manual; (2) Summary Transaction Group, in. which subjects studied from a manual 
that included a verbal summary of the key actions carried out for each BASIC 
statement; C3) Transaction Group, in which subjects studied from a' manual that 
included a verbal description of each action carried oUt by ,the computer for ^^.^^ 
each BASIC statement; (A) Diagram Group, in which subjects studied from a 
manual that included diagrams showing each of the actions carried out by the 

er|c ' ' 22u 
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computer, for BAStC statement^; (5) Transaction/Diagram Group, in which 

subjects studied a manual that cbntalned both a verbal description and a 
diagram description for each BASIC statement. 

There were 19 subjects' In each group; however, data from one subject In 
Group 5 was eliminated l>ecause that subject failed to follow directions. 

Materials . The materials consisted of a subject questionnaire, five 
Instructional manuals » two mastery tests, and four posttests. 

The subject questionnaire was typed onto an 8^5 x 11 Inch sheet of paper » • 
and consisted ot questions concerning: the subject's demographic 
characteristics, such as age, sex, and academic major; the subject'a 
mathematics background » such as geometry, algebra and calculus courses the 
individual had taken in high. school or college; the subject's ability, such as 
SAT-Verbal score, SAT-Quantitative score, and grade point average; the 
subject's programming background, such as whether the individual' had ever 
operated a programmable calculator, typed at a computer terminal, etc. 

There were five kinds of manuals: standard manual, summary transaction 
manual, transaction manual, diagram manual* and transact ion/dia;gram manual.' 
Each manual was typed double-spaced, onto 8^ x 11 inch sheets of paper. 

The standard manual was adopted from a self-instructional, mastery manual 
called BASIC in Six Hot3irs (Marcus, 1980) which is widely used in teaching ' 
BASIC to students in the Microcomputer Laboratory of the University of 
California, Santa Barbara. The manual consisted of two lessons. The first 
lesson covered statement execution in the Immediate mode as well 'as some 
introductory information for preparing a program. The statements and commands 
introduced in the first lesson were: PRINT, LET, IF-THEN statements and NEW, 
LIST and RUN commands. The second lesson covered programming and included . 
some sample programs. The statements introduced in the second lesson were 
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GOTO, IF-GOTO, INPUT, READ and DATA.. The standard manual provided Information 
concerning the syntax for. each statement; examples provided ~ information 
concerning what the output would be for a given input. For example, PRINT X 
results in a. number appearing on the screen. 

The summary transaction manual was identical to the standard manual 
except for some added information. For each Statement, a description of the 
key actionfe'that occur within the computer was given. For example, 
LET X « A + B me^ns that computer "stores in space X the sum of the number In 
A and- the number' in B." 

The transaction manual was idfntical to the standard manual except for 
some added information. For- each statement, a detailed description of each 
action that occurs within the computer was given. For example, LET X ■ A + B, 
means: "find the number in space A but do not erase it," "find the number in 
memory space B but do not erase it," "add those numbers together," "erase the 
old number in* space XV "put the sum in space X," "go on to the next 
statement," 

The diagram manual was identical to the standard manual except that some 
information was added. For each. statement, a' diagram of, the inside of the 
computer was shown before and after the statement was executed. The diagram 
showed the status of an input stack, an output screen, a wai^-»run light, a 
program list, ^d a memory scoreboard* For example, fo* LET X « A + B, in %he 
before diagram the number 5 was in space A, 4 in B and 3 in X; the after 
diagram showed 5 in A, 4 in B, and 9 in X. 

The ti;^ansaction/diagram manual was the same as the transaction maniial and 
diagram manuals combined. For each statement both a verbal' description of 
action and before/after diagrams were^ given. 
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There were two sets of loastery -tests — one for lesson 1 and one for lesson 
2. Each test was typed on a 8% x 11 Inch sheet of paper. Each test consisted 
of 5 questions based on the contents of the lesson* Each test h^d a 10*ialnute 
time limit* Several versions of each mastery test were available for eachr 
lesson. . 

There were four posttest^: general criterion, fact, verbal 
Specification, diagram specification. The general criteria that consisted of 
18 problems, each typed on a 5 x 8 Inch sheet. Including six of each of the 
following types: Interpretation problems presented a statement or program and 
asked the subject to write what was accomplished In English; generation 
problems presented a problem In English and asked the subject to write a 
statement or program to solve the problem; debugglpg problems presented a 
flawed statement or program and asked the subject to tell what was wrong. 

The fact retention test consisted of twenty f ill-ln-the-blanks questions 
about Information that was explicitly stated In the manuals. The test was 
typed ou an 8^5 x 11 inch sheet of paper. \ 

The verbal specification task test consisted of ten 8% x 11 Inch, sheets 
of paper, with a BASIC statement typed on each. The subject's job -^'s to • 
write. In plain English, the actions that the computer would carry out to 
execute each statement. Each sheet contained sevetal blank lines, and 

■ 1 

subjects were Instructed to write »one action per line. The following 
statements were given: 70 LET a - B + 1; 60 PATA 5, 10, 13; 10 PRINT C; 
30 £ET D - 0; 80 GOTO 10; 20 READ B; 90 PRINT 'C"; 40 IF A > B THEN GOTO 99; 
100 INPUT A; -50 IF A-B^THEN PINT "THEY ARE EQUAL". 

The diagram Specification test consisted of ten 8*5 x 11 inch pages, with 

« . ' 

t 

a BASIC statement typed on each. In addition, the top of the page presented 
In diagram of the computer indication the contents of the Input stack, the 
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contents of the memory scoreboard » the status of the run-wait lights the 
information printed on the output screen^ and the pointer arrow's location in 
the program list. This diagram represented the state of^he computer before 
the statement was executed. At the bottom of the page, there was a blank 
diagram for the subject to fill in in order to Indicated the status of the 
computer after the statement is executed. The same 10 statements were used as 
for the verbal specification test. 

Procedure. The experiment consisted of five onfe-hour sessions over a one 
to two weeU period. In the first session, subjects filled out the subject 
questionnaire, received a brief description of the study, and signed up for 
times for the following four sessions. In the second and third session, 
subjects worked on their respective manuals, folj-owing random a'siignment for 
treatment groups. Subjects studied the manuals at their own pace, one le«^on 
at . a session. Subjects were given the 10-minute mastery test after each * 
lesson. If a subject missed more than otie item, thJfe errors were explained by 
the experimenter and the subject was asked to review |:he manual. Then, 
anotlier form of the mastery test was given, and so on. During the final two 
sessions, the four poat tests were given in the following order: general 
criterion, fact retention, verbal specification, diagram specification. 
Subjects answered .the questions on the tests at thfeir own pace. 
Results 

This section provides a preliminary analysis of the results with special 
focus on whether the instifuctional treatments (I.e., groups 2 through 5) 
increased problem solving performances (criterion test) and decreased 
misconceptions (verbal and diagram speckf ic^tlon tests) . I 

Do the groups differ In study tfmei Since the experimental manuals 

5 1 



(Groups 2 through 5) contain more material than the standard manual, it la, 
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likely .that subjects in the experimental groups will require more study time. 
The mean times required to read the-two lessons for Groups 1 through 5 were 
43 » 48 » 49 » 59 and 74 minutes » respectively. An analysis of variance revealed 
significant dif f erencfifs among the groups, F(4,89) « 15.24, p < .001. 

Does explicit model instruction affect problem solving? Learning a 
mental model should enhance performance on the criterion test. The average 
correct on the criterion test was computed for each of the five treatment , * 
groups. An analysis of variance revealed no significant differences among the 
groups In criterion score, F^<^1. Individual t-tests arevealed no significant 
differences ^mong the four ex]|>erimental groups (i.e.. Groups' 2, 3, 4 and 5)^^ 
therefore, these four groups were combined for subsequent analysis. The 
subjects in each groii^ were divided into two ability levels — those scoring 
above 530 in SAT-Quantative were called "high ai[)ility" and those scoring at or 
below 530' were called "low ability." An analysis of variance was conducted 
with treatment (group 1 versus the combination of all experimental groups) and 
ability (high versus low) as between subject factors. The ANOVA revealed a 
sigi}lficant aptitude treatment interaction (ATI) in which low ability students 
learned better when given explicit training in mental models (experimental 
groups) but high ability learners did better when given only -the standard 
manual (group 1), B^(l,90) « 3.84, p < .05. Apparently, high ability leari^ers 
are able to generate their own useful models based on their past experience in 
mathematics tasks while low ability learners benefit from the instructor 
explicitly providing a model. These re^sults are summarized in Table 1. 



Insert Table 1 about here 
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Do the groups differ in retention of specific facts? Leai(nl«g a mental 
model should not affect the learner's retention of specific fact^, since the 
model is independent of tl;ie specific facts. ' Table 2 lists the percentage 
correct retention for the five groups. As can be se^n^ the experimental 
treatment does not enhance fact retention. An analysis of variance revealed 
no significant differences among the means» F(4»89) f l.TA* 

J Insert Table 2 about here 



Does explicit training in mental models reduce the niyber of 
mis concep t ions ? The experimental treatments (groups 2 through 5) should 
result in fewer misconceptions than the Qtandaifd treatment In order to test 
this Idea,^ the verbal and diagram specification tests were scored. If the^ 
user generated the key 'actions, and no incorrect actions, for a statement^ 
that statement was counted as correct. Table 3 shows the mean number of 
correct statements for subjects in ea ch o f the five treatment groups. As can 
be seen, the experimental groups tend to^^perfor^r^-be^^ than the standard 
group. An analysis of variance comparing the standard group against the four 
experimental groups revealed a significant difference in overall number of 
correct statements, F (1, 89) » 5.84, p < .05. 



Insert Table 3 about here 



Discussion 

This study provides promising indications that nental models can be 
explicitly taught to novices. In particular » explicit teaching of mental 
models seems inost effective for low ability learners. This result isP 



Si- * . 

\ consistent with earlier findings by Mayer (1975). In addition, the 

t^t;.*^ , • * transaction treatment (group 3) seems to be the most efficient Instructional 
^ • procedure because It requires very little additions,! time while producing the 

largest gains In problem solving -and verbal specification performance. This 
8tu5y compli,ment8 earlier research (Bayman & Mayer, In press) showing that 
novices tended to acquire many misconceptions. Apparently, fairly modest 
Z-^. * alterations In the Instructional prograir (e.g., adding a discussion of 

/'H-'-'H^ ( , ' ' . ' 

^ "transactions) can gteatly reduce user misconceptions. The reduction of 

• ^^^ t ^ 

'-^conceptions Is especially Important for users who plan to continue learning 



' ' ' about computer programming, since they will need to build on their existing 
^ronSeptlons. Further work is needed to determine the long-term effects of ' 



user 'misconceptions, and the effects of remediation. 
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Table 1., , 

Percent Correct on the ^General Criterion Test for Experimental 

and Standard Subjects 

Treatment Groups -\ ASlllty Level 

Low High , 

Experimental Groups * 58Z 65% 

Standard Group ' ,48% 74% ' 



% 
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, Table 2 

Percentage Correct on the Fact Retention Test for« Five Crpups 

Treatment Groub . ' Pe rcent Correct 

Standard ■ ^ 12% ^ 

Suranary Transaction ^ S 7435 

Transaction ' . 70% 

Diagram 62% 

Transact lon/D/agram ,72% 
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Table 3 

Percent of statemeigits. Described Correctly by Subjects in 
* * Five Treatment Groups 

Tr^tment Group Percent Correct 

( ' 

Standard 58% 

Summary Transaction 64% 

Transaction 71% 

Diagram - 66% 

Tran*ictlon/Dlagram 66% 
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